Nov 01 15:44:59.163382 ip-10-0-141-145 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 01 15:44:59.163394 ip-10-0-141-145 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 01 15:44:59.163417 ip-10-0-141-145 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 01 15:44:59.163672 ip-10-0-141-145 systemd[1]: Failed to start Kubernetes Kubelet. Nov 01 15:45:09.236359 ip-10-0-141-145 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 01 15:45:09.236382 ip-10-0-141-145 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot f8cba4afe90c4cf5885d4faf94a6d338 -- Nov 01 15:47:00.922084 ip-10-0-141-145 systemd[1]: Starting Kubernetes Kubelet... Nov 01 15:47:01.384700 ip-10-0-141-145 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:01.384700 ip-10-0-141-145 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 15:47:01.384700 ip-10-0-141-145 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:01.384700 ip-10-0-141-145 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:01.389689 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389597 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:01.389689 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389683 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:01.389689 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389689 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:01.389689 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389694 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389698 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389703 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389707 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389711 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389716 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389720 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389723 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389727 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389731 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389735 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389739 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389743 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389763 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389769 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389773 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389777 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389781 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389786 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:01.389961 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389790 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389794 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389798 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389802 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389814 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389818 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389822 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389826 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389830 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389835 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389839 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389844 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389851 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389858 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389863 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389870 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389875 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389880 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389885 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:01.390738 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389889 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389894 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389898 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389902 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389907 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389911 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389915 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389920 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389924 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389929 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389933 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389937 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389943 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389947 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389951 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389956 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389959 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389964 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389968 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:01.391528 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389972 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389977 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389981 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389985 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389990 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389994 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.389998 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390003 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390007 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390012 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390017 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390023 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390027 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390032 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390038 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390044 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390049 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390053 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390058 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:01.392453 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390063 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390067 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390072 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390076 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390081 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390085 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390090 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390095 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390099 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390103 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390108 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390112 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390116 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390120 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390125 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390129 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390134 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390137 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390141 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390145 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:01.393336 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390150 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390154 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390161 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390167 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390172 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390176 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.390181 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391518 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391529 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391534 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391538 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391543 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391547 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391551 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391555 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391560 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391564 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391568 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391573 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391577 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:01.394205 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391582 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391587 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391591 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391598 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391604 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391609 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391614 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391618 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391623 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391627 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391632 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391637 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391641 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391645 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391651 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391655 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391660 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391665 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391669 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:01.395101 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391674 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391678 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391682 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391686 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391691 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391695 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391699 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391703 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391708 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391712 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391717 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391721 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391725 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391730 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391735 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391741 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391765 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391770 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391776 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:01.395907 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391781 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391785 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391790 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391794 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391799 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391803 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391807 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391812 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391817 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391821 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391826 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391831 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391835 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391839 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391843 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391848 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391852 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391857 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391861 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391865 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:01.396432 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391869 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391874 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391878 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391882 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391886 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391890 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391894 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391898 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391904 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391908 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391912 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391917 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391921 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391925 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391929 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391933 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391937 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391941 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391947 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391951 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:01.398025 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391956 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391961 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391965 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391968 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391973 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391977 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391981 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391986 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391990 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391995 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.391999 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392004 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392008 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392012 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392016 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392840 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392850 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392854 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392859 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392866 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:01.398505 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392872 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392877 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392882 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392886 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392891 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392895 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392899 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392903 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392908 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392913 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392917 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392921 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392927 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392932 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392937 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392941 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392945 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392950 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392954 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:01.399023 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392959 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392963 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392968 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392972 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392976 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392980 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392984 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392988 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392992 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.392996 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393001 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393005 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393009 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393013 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393018 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393022 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393026 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393030 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393035 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393038 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:01.399471 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393043 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393047 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393053 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393057 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393061 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393068 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393077 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393082 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393086 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393090 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393094 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393098 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393103 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393107 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393111 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393115 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393120 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393124 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:01.399962 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393131 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393138 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393144 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393149 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393154 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393158 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393162 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393166 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393171 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393176 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393180 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393184 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393188 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393193 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393198 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393202 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393207 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393211 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393218 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:01.400860 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393222 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393226 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393231 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393235 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393239 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393243 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393247 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393251 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393256 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393260 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393264 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393269 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393273 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393277 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393281 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393285 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393289 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393293 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393297 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393301 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:01.401335 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393306 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393311 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393317 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.393324 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393476 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393493 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393502 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393509 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393516 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393521 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393528 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393538 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393543 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393548 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393554 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393559 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393564 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393569 2579 flags.go:64] FLAG: --cgroup-root="" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393574 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393579 2579 flags.go:64] FLAG: --client-ca-file="" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393583 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393588 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393596 2579 flags.go:64] FLAG: --cluster-domain="" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393600 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 15:47:01.401828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393605 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393611 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393617 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393623 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393628 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393633 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393638 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393643 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393648 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393654 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393659 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393664 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393671 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393677 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393681 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393686 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393691 2579 flags.go:64] FLAG: --enable-server="true" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393698 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393705 2579 flags.go:64] FLAG: --event-burst="100" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393712 2579 flags.go:64] FLAG: --event-qps="50" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393717 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393722 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393727 2579 flags.go:64] FLAG: --eviction-hard="" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393734 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 15:47:01.402383 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393739 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393744 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393764 2579 flags.go:64] FLAG: --eviction-soft="" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393770 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393774 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393779 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393784 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393789 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393794 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393798 2579 flags.go:64] FLAG: --feature-gates="" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393805 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393810 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393815 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393821 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393826 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393831 2579 flags.go:64] FLAG: --help="false" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393835 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393841 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393846 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393851 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393857 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393863 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393868 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 15:47:01.403028 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393873 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393877 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393884 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393889 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393896 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393901 2579 flags.go:64] FLAG: --kube-reserved="" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393906 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393911 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393916 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393920 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393926 2579 flags.go:64] FLAG: --lock-file="" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393930 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393935 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393941 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393950 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393955 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393960 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393964 2579 flags.go:64] FLAG: --logging-format="text" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393969 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393975 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393980 2579 flags.go:64] FLAG: --manifest-url="" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393984 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393991 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.393996 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394002 2579 flags.go:64] FLAG: --max-pods="110" Nov 01 15:47:01.403589 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394007 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394012 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394017 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394022 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394027 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394033 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394038 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394050 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394055 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394062 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394067 2579 flags.go:64] FLAG: --pod-cidr="" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394073 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394078 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394083 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394088 2579 flags.go:64] FLAG: --port="10250" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394093 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394098 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-030866921209e4fe1" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394103 2579 flags.go:64] FLAG: --qos-reserved="" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394108 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394113 2579 flags.go:64] FLAG: --register-node="true" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394117 2579 flags.go:64] FLAG: --register-with-taints="" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394123 2579 flags.go:64] FLAG: --registry-burst="10" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394128 2579 flags.go:64] FLAG: --registry-qps="5" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394133 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394138 2579 flags.go:64] FLAG: --reserved-memory="" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394144 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 15:47:01.404210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394149 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394154 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394159 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394163 2579 flags.go:64] FLAG: --runonce="false" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394168 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394174 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394179 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394184 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394189 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394194 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394199 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394203 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394209 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394213 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394218 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394224 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394229 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394236 2579 flags.go:64] FLAG: --system-cgroups="" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394241 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394250 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394255 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394260 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394267 2579 flags.go:64] FLAG: --tls-min-version="" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394271 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394276 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 15:47:01.404848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394281 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394286 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394290 2579 flags.go:64] FLAG: --v="2" Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394297 2579 flags.go:64] FLAG: --version="false" Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394304 2579 flags.go:64] FLAG: --vmodule="" Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394310 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.394315 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394554 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394561 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394566 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394570 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394575 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394579 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394583 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394588 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394592 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394597 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394602 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394606 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394610 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394616 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:01.405431 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394620 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394626 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394631 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394636 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394641 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394645 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394650 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394665 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394671 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394676 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394680 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394684 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394689 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394693 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394697 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394702 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394707 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394711 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394716 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:01.405958 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394720 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394724 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394728 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394732 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394737 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394741 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394745 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394765 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394770 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394774 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394779 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394783 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394788 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394793 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394800 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394807 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394814 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394819 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394823 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:01.406424 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394827 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394832 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394836 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394840 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394845 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394849 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394853 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394857 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394862 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394867 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394871 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394875 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394880 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394884 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394889 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394893 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394897 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394901 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394906 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:01.406895 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394910 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394915 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394919 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394923 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394928 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394932 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394936 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394941 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394948 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394952 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394958 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394963 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394967 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394971 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394976 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394980 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394985 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394989 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394993 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.394997 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:01.407374 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395001 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395006 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395011 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395015 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395019 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395023 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395027 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395032 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395036 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395040 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395044 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395049 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395053 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395058 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395062 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.395103 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:01.407875 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395167 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395176 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395182 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395187 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395192 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395200 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395205 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395210 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395214 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395218 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395222 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395227 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395232 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395236 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395240 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395244 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395249 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395253 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395258 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:01.408250 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395262 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395266 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395271 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395276 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395280 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395284 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395288 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395293 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395297 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395301 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395305 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395310 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395314 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395319 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395323 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395327 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395332 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395336 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395342 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395347 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:01.408695 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395352 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395356 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395360 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395365 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395375 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395379 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395384 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395388 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395393 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395397 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395401 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395405 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395410 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395414 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395419 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395423 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395427 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395432 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395440 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:01.409231 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395446 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395451 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395455 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395459 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395464 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395468 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395472 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395477 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395482 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395486 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395491 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395497 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395501 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395506 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395511 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395515 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395519 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395524 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395528 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395533 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:01.409668 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395537 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395541 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395545 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395550 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395554 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395558 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395565 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395571 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395575 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395580 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395585 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395589 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395593 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395598 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395602 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395606 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395611 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395615 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395619 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:01.410153 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395623 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395628 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395632 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395637 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395643 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395647 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395652 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395657 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.395661 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.402348 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.402362 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.404404 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.404411 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404466 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404472 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404476 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404479 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404482 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404485 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:01.410592 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404488 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404490 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404493 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404496 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404499 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404501 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404504 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404508 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404510 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404513 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404516 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404518 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404522 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404524 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404527 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404530 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404533 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404535 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404538 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:01.411062 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404541 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404544 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404547 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404550 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404553 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404556 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404559 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404562 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404565 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404567 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404570 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404573 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404577 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404581 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404584 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404588 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404591 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404594 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404597 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:01.411513 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404600 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404603 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404606 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404608 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404611 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404614 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404617 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404620 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404622 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404625 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404628 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404630 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404633 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404636 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404638 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404641 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404645 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404650 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404653 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:01.412043 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404657 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404660 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404662 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404665 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404668 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404671 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404674 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404676 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404679 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404681 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404684 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404687 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404690 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404692 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404695 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404698 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404700 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404703 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404706 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404709 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:01.412784 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404712 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404715 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404718 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404720 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404723 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404725 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404728 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404731 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404733 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404737 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404740 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404742 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404745 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404761 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404765 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404767 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404770 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404773 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404775 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404778 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:01.413440 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404781 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404783 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404786 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.404814 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404967 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404972 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404975 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404978 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404981 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404983 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404986 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404989 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404992 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404995 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.404997 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405000 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:01.414034 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405002 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405005 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405008 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405010 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405013 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405016 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405019 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405022 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405025 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405027 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405030 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405032 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405035 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405038 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405040 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405043 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405046 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405048 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405051 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405053 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:01.414441 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405056 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405058 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405061 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405063 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405066 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405068 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405071 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405074 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405076 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405079 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405082 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405084 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405086 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405089 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405092 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405115 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405119 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405122 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405125 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:01.414940 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405128 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405131 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405134 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405136 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405139 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405142 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405144 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405146 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405149 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405152 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405154 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405157 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405161 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405164 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405168 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405172 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405175 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405177 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405180 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:01.415378 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405182 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405185 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405188 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405191 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405193 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405196 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405198 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405201 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405204 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405206 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405209 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405211 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405214 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405217 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405220 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405223 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405225 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405228 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405231 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405233 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:01.415850 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405236 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405238 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405241 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405243 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405246 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405248 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405251 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405253 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405256 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405258 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405261 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405263 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405266 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405269 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405271 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:01.416310 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405274 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.405297 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405328 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405332 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405336 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405339 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405342 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405345 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405348 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405353 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405355 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405358 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405361 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405364 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405367 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405369 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:01.416667 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405372 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405374 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405377 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405380 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405382 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405385 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405387 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405390 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405392 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405395 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405397 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405400 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405403 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405405 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405408 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405410 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405413 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405415 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405418 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:01.417112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405421 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405423 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405425 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405428 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405430 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405433 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405437 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405440 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405443 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405446 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405448 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405451 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405454 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405456 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405459 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405462 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405464 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405467 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405470 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:01.417550 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405472 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405475 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405477 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405480 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405482 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405485 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405487 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405490 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405492 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405495 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405498 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405500 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405503 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405505 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405507 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405510 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405513 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405516 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405518 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405522 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:01.418005 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405525 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405528 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405530 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405533 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405535 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405538 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405541 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405543 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405546 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405548 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405551 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405553 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405556 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405558 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405561 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405564 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405566 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405569 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405571 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:01.418465 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405575 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405578 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405581 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405584 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405587 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405591 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405594 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405598 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405601 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405605 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405608 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405610 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405614 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405617 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:01.405619 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.405668 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.408077 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.409045 2579 server.go:1014] "Starting client certificate rotation" Nov 01 15:47:01.418930 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.409517 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:47:01.419344 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.409550 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:47:01.419344 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.411041 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:01.419344 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.413725 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:01.428049 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.428032 2579 log.go:25] "Validated CRI v1 runtime API" Nov 01 15:47:01.431883 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.431867 2579 log.go:25] "Validated CRI v1 image API" Nov 01 15:47:01.432728 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.432703 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 15:47:01.439440 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.439416 2579 fs.go:135] Filesystem UUIDs: map[0c52f548-7fd8-4640-965b-ca34386c28a7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fde469a6-9826-4d71-8302-4d99deeb5887:/dev/nvme0n1p3] Nov 01 15:47:01.439517 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.439438 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 15:47:01.445272 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.445165 2579 manager.go:217] Machine: {Timestamp:2026-11-01 15:47:01.443301063 +0000 UTC m=+0.392118980 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102966 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e78f05160518db3a72d2f7df20f2 SystemUUID:ec28e78f-0516-0518-db3a-72d2f7df20f2 BootID:f8cba4af-e90c-4cf5-885d-4faf94a6d338 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:af:df:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:af:df:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:39:e0:08:f0:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 15:47:01.445272 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.445271 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 15:47:01.445372 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.445345 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 15:47:01.448742 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.448721 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 15:47:01.448884 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.448743 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-145.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 15:47:01.449451 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.449433 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 15:47:01.449529 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.449521 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 15:47:01.449568 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.449534 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 15:47:01.449568 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.449546 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:01.449568 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.449556 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:01.450467 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.450457 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 15:47:01.454057 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.454044 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 15:47:01.454182 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.454172 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 15:47:01.454218 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.454206 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 01 15:47:01.454218 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.454216 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 15:47:01.454271 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.454238 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 15:47:01.454271 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.454247 2579 kubelet.go:405] "Adding apiserver pod source" Nov 01 15:47:01.454271 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.454256 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 15:47:01.454593 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.454582 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:01.458615 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.458594 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 15:47:01.463730 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.463717 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:01.464494 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.464481 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 15:47:01.464539 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.464512 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 15:47:01.464673 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.464664 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 15:47:01.464702 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.464679 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 15:47:01.464702 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.464686 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 15:47:01.464702 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.464692 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 15:47:01.464702 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.464698 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 15:47:01.464702 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.464703 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 15:47:01.465267 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.465259 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 15:47:01.465295 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.465268 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 15:47:01.465295 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.465275 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 15:47:01.465295 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.465281 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 15:47:01.465295 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.465290 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 15:47:01.465295 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.465298 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 15:47:01.466677 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.466661 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m52kg" Nov 01 15:47:01.467021 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.467010 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 15:47:01.467057 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.467026 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 15:47:01.470670 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.470647 2579 server.go:1263] "Started kubelet" Nov 01 15:47:01.470775 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.470696 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 15:47:01.470853 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.470803 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 15:47:01.470905 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.470874 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 15:47:01.471504 ip-10-0-141-145 systemd[1]: Started Kubernetes Kubelet. Nov 01 15:47:01.472890 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.472862 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 01 15:47:01.474350 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.473916 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m52kg" Nov 01 15:47:01.474453 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.474436 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-145.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 01 15:47:01.474508 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.474447 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-145.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 01 15:47:01.475481 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.475451 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 01 15:47:01.475552 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.475513 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 15:47:01.475994 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.475766 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 15:47:01.480375 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.480357 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 15:47:01.480375 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.480368 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 15:47:01.480583 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.480421 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:01.481032 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481014 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 15:47:01.481099 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481086 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 15:47:01.481144 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481101 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 15:47:01.481222 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481204 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 01 15:47:01.481222 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481218 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 01 15:47:01.481431 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481409 2579 factory.go:55] Registering systemd factory Nov 01 15:47:01.481506 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481454 2579 factory.go:223] Registration of the systemd container factory successfully Nov 01 15:47:01.481783 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481768 2579 factory.go:153] Registering CRI-O factory Nov 01 15:47:01.481855 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481785 2579 factory.go:223] Registration of the crio container factory successfully Nov 01 15:47:01.481855 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481844 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 15:47:01.481961 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481868 2579 factory.go:103] Registering Raw factory Nov 01 15:47:01.481961 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.481893 2579 manager.go:1199] Started watching for new ooms in manager Nov 01 15:47:01.482298 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.482266 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-145.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 01 15:47:01.482951 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.482939 2579 manager.go:319] Starting recovery of all containers Nov 01 15:47:01.483408 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.482394 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-145.us-east-2.compute.internal.18e3f66c435c5de6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-145.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-145.us-east-2.compute.internal,},FirstTimestamp:2026-11-01 15:47:01.470617062 +0000 UTC m=+0.419434979,LastTimestamp:2026-11-01 15:47:01.470617062 +0000 UTC m=+0.419434979,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-145.us-east-2.compute.internal,}" Nov 01 15:47:01.495500 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.494819 2579 manager.go:324] Recovery completed Nov 01 15:47:01.497459 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.497436 2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Nov 01 15:47:01.499701 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.499683 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:01.500373 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.500361 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:01.502305 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.502286 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:01.502398 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.502324 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:01.502398 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.502337 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:01.502886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.502872 2579 cpu_manager.go:226] "Starting" policy="none" Nov 01 15:47:01.502886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.502883 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 15:47:01.502982 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.502899 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 15:47:01.502982 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.502919 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:01.503591 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.503575 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:01.503633 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.503603 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:01.503633 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.503613 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:01.563153 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.554352 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 15:47:01.563153 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.555519 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 15:47:01.563153 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.555539 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 15:47:01.563153 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.555583 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 15:47:01.563153 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.555624 2579 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 15:47:01.563153 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.559060 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:01.640109 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.581517 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:01.640109 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.638905 2579 policy_none.go:50] "Start" Nov 01 15:47:01.640109 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.638929 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 15:47:01.640109 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.638948 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 15:47:01.641033 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.641019 2579 policy_none.go:44] "Start" Nov 01 15:47:01.656299 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.656278 2579 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 15:47:01.677475 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.677284 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 15:47:01.681443 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.681428 2579 manager.go:341] "Starting Device Plugin manager" Nov 01 15:47:01.681606 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.681590 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:01.681966 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.681951 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 15:47:01.682007 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.681966 2579 server.go:87] "Starting device plugin registration server" Nov 01 15:47:01.682223 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.682209 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 15:47:01.682292 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.682229 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 15:47:01.682353 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.682291 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 15:47:01.682411 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.682363 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 15:47:01.682411 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.682374 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 15:47:01.683056 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.683037 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 15:47:01.683117 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.683088 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:01.689016 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.689002 2579 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-145.us-east-2.compute.internal\" not found" node="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.782456 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.782420 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:01.783434 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.783418 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:01.783496 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.783449 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:01.783496 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.783462 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:01.783496 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.783487 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.791957 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.791934 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.792046 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.791960 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-145.us-east-2.compute.internal\": node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:01.804657 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.804630 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:01.857188 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.857152 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal"] Nov 01 15:47:01.857305 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.857230 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:01.858313 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.858297 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:01.858370 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.858335 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:01.858370 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.858347 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:01.860730 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.860717 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:01.860877 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.860860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.860914 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.860895 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:01.861597 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.861580 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:01.861668 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.861611 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:01.861668 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.861624 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:01.861668 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.861584 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:01.861782 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.861679 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:01.861782 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.861702 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:01.863858 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.863843 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.863928 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.863870 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:01.864616 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.864600 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:01.864669 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.864633 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:01.864669 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.864658 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:01.882601 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.882580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.882714 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.882608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.882714 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.882631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.884735 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.884718 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" node="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.891635 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.891590 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" node="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.905116 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:01.905089 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:01.983345 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.983322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.983447 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.983352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.983447 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.983373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.983447 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.983432 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.983535 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.983486 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:01.983535 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:01.983520 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:02.005694 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:02.005669 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:02.106226 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:02.106200 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:02.187705 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.187670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:02.195311 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.195291 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:02.206654 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:02.206632 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:02.307126 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:02.307097 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:02.407570 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:02.407528 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:02.408633 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.408615 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 15:47:02.408798 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.408782 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 15:47:02.408835 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.408796 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 15:47:02.481898 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.481820 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 15:42:01 +0000 UTC" deadline="2028-06-08 01:18:00.454818192 +0000 UTC" Nov 01 15:47:02.481898 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.481860 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14025h30m57.972961167s" Nov 01 15:47:02.508328 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:02.508293 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:02.608745 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:02.608712 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-145.us-east-2.compute.internal\" not found" Nov 01 15:47:02.615273 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.615247 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:02.681237 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.681209 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:02.691302 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.691275 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:02.692917 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.692895 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:02.700778 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.700763 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:02.826101 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.826071 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:02.858867 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:02.858833 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92683d17f4ef39773bb8bc1b1459025a.slice/crio-ba4251a2aab9ac5f477e533e1061c75fe685c92bc68877976888934f220d0679 WatchSource:0}: Error finding container ba4251a2aab9ac5f477e533e1061c75fe685c92bc68877976888934f220d0679: Status 404 returned error can't find the container with id ba4251a2aab9ac5f477e533e1061c75fe685c92bc68877976888934f220d0679 Nov 01 15:47:02.859190 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:02.859170 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6555262b15c85369c5c728ca31cfdef1.slice/crio-10f5ee1b941f2c53e73e955e1092ccfca4a2a2ff935b6c2763c7e1100b244af0 WatchSource:0}: Error finding container 10f5ee1b941f2c53e73e955e1092ccfca4a2a2ff935b6c2763c7e1100b244af0: Status 404 returned error can't find the container with id 10f5ee1b941f2c53e73e955e1092ccfca4a2a2ff935b6c2763c7e1100b244af0 Nov 01 15:47:02.863167 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:02.863154 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:47:03.410291 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.410252 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:03.455460 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.455423 2579 apiserver.go:52] "Watching apiserver" Nov 01 15:47:03.468585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.468514 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 15:47:03.469516 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.469444 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-bdz9j","kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal","openshift-dns/node-resolver-8fxdr","openshift-image-registry/node-ca-rsjf4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal","openshift-network-operator/iptables-alerter-q52bf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw","openshift-network-diagnostics/network-check-target-2xm97","openshift-ovn-kubernetes/ovnkube-node-x4htf","kube-system/global-pull-secret-syncer-2g5v9","kube-system/konnectivity-agent-cvct9","openshift-cluster-node-tuning-operator/tuned-p66vf","openshift-multus/multus-additional-cni-plugins-bmsrc","openshift-multus/multus-b6ttz"] Nov 01 15:47:03.471965 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.471944 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:03.472495 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.472476 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.474713 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.474663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:03.476637 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.476619 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.477114 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.477099 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 15:47:03.477114 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.477106 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 15:47:03.477244 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.477225 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 15:47:03.477302 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.477243 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5vhjr\"" Nov 01 15:47:03.477357 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.477332 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 15:47:03.477406 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.477368 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-28hsn\"" Nov 01 15:47:03.477475 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.477460 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 15:47:03.479096 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.479076 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 15:47:03.479191 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.479154 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 15:47:03.481029 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.480874 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 15:47:03.481029 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.480988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.482737 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.482715 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bwtj5\"" Nov 01 15:47:03.483746 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.483729 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 15:47:03.483923 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.483735 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 15:47:03.484022 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.483832 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9hxd8\"" Nov 01 15:47:03.484022 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.483840 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 15:47:03.484455 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.484438 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.486918 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.486899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.489026 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.488908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.490991 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.490968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-multus-certs\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491081 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-socket-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.491081 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491028 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-etc-selinux\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.491081 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491056 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djclq\" (UniqueName: \"kubernetes.io/projected/edc57054-ce41-4e4e-97f6-b127bf151d23-kube-api-access-djclq\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-system-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-socket-dir-parent\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-device-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22ea5ba-32e9-4552-802d-d11594d04f29-tmp-dir\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b7q6\" (UniqueName: \"kubernetes.io/projected/a22ea5ba-32e9-4552-802d-d11594d04f29-kube-api-access-7b7q6\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc57054-ce41-4e4e-97f6-b127bf151d23-host\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-k8s-cni-cncf-io\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-netns\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491240 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-kubelet\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-etc-kubernetes\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cni-binary-copy\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-daemon-config\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-registration-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cnibin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-bin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-multus\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edc57054-ce41-4e4e-97f6-b127bf151d23-serviceca\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491436 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-hostroot\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-sys-fs\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgr87\" (UniqueName: \"kubernetes.io/projected/06506c1c-88b0-42e1-ab42-8cc842decd18-kube-api-access-rgr87\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-conf-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd7gp\" (UniqueName: \"kubernetes.io/projected/e8a2984b-e706-43f8-bd87-dd5bc809c36e-kube-api-access-dd7gp\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7e5339a-b073-4184-9acc-5a701276ea1c-agent-certs\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:03.491662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7e5339a-b073-4184-9acc-5a701276ea1c-konnectivity-ca\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:03.492517 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a22ea5ba-32e9-4552-802d-d11594d04f29-hosts-file\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.492517 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.491669 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.492517 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.492162 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 15:47:03.493564 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.493548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:03.493650 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.493610 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:03.496388 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.496370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.496480 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.496437 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:03.498815 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.498796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:03.498913 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.498858 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:03.501199 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.501183 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.509901 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.509877 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkp4h" Nov 01 15:47:03.516829 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.516800 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkp4h" Nov 01 15:47:03.561828 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.561744 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"ba4251a2aab9ac5f477e533e1061c75fe685c92bc68877976888934f220d0679"} Nov 01 15:47:03.564827 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.564803 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"6555262b15c85369c5c728ca31cfdef1","Type":"ContainerStarted","Data":"10f5ee1b941f2c53e73e955e1092ccfca4a2a2ff935b6c2763c7e1100b244af0"} Nov 01 15:47:03.577677 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.577642 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 15:47:03.577852 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.577707 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:03.577852 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.577811 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-58l9j\"" Nov 01 15:47:03.577965 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.577856 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 15:47:03.577965 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.577872 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 15:47:03.578082 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.577965 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 15:47:03.578082 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578005 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:03.578082 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578019 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 15:47:03.578082 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578033 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 15:47:03.578679 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578656 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjc9m\"" Nov 01 15:47:03.578786 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578695 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 15:47:03.578786 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578770 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 15:47:03.578905 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578823 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ttxgn\"" Nov 01 15:47:03.578905 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578865 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 15:47:03.579013 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578915 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 15:47:03.579013 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.578954 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 15:47:03.579104 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.579051 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jsz89\"" Nov 01 15:47:03.579191 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.579179 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:03.579268 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.579249 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 15:47:03.579369 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.579316 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:03.579422 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.579385 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6x5r5\"" Nov 01 15:47:03.579471 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.579443 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 15:47:03.582359 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.582343 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 15:47:03.591834 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.591807 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-systemd-units\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.591925 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.591844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-conf\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.591925 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.591871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rzcv\" (UniqueName: \"kubernetes.io/projected/ada3c550-b52f-4236-88db-a07ed2c6fdda-kube-api-access-6rzcv\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.592004 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.591944 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s446\" (UniqueName: \"kubernetes.io/projected/0a2d0642-db9c-4bd7-a804-51167fdd62fd-kube-api-access-4s446\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:03.592004 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.591987 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-bin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.592084 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-multus\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.592084 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-systemd\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.592164 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592091 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-sys\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.592164 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592114 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-host\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.592259 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-dbus\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.592259 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-hostroot\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.592259 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-sys-fs\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.592259 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-netns\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.592436 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.592436 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592288 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-cnibin\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.592436 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.592436 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-kubelet-config\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.592436 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.592436 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd7gp\" (UniqueName: \"kubernetes.io/projected/e8a2984b-e706-43f8-bd87-dd5bc809c36e-kube-api-access-dd7gp\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a22ea5ba-32e9-4552-802d-d11594d04f29-hosts-file\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592476 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-netd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysconfig\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-multus-certs\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-socket-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-config\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-ovn\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-bin\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-script-lib\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.592709 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592681 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-modprobe-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-socket-dir-parent\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-device-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-etc-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc57054-ce41-4e4e-97f6-b127bf151d23-host\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592926 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-k8s-cni-cncf-io\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592950 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-kubelet\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.592976 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-etc-kubernetes\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ada3c550-b52f-4236-88db-a07ed2c6fdda-iptables-alerter-script\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-log-socket\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cni-binary-copy\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-kubelet\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-systemd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593140 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-node-log\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-kubernetes\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cnibin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.593225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-tuned\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada3c550-b52f-4236-88db-a07ed2c6fdda-host-slash\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-system-cni-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593336 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwkgh\" (UniqueName: \"kubernetes.io/projected/4a228ff8-0200-4b6f-8948-9e351926a3b7-kube-api-access-vwkgh\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593361 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593384 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-tmp\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edc57054-ce41-4e4e-97f6-b127bf151d23-serviceca\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgr87\" (UniqueName: \"kubernetes.io/projected/06506c1c-88b0-42e1-ab42-8cc842decd18-kube-api-access-rgr87\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593560 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-env-overrides\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cksfz\" (UniqueName: \"kubernetes.io/projected/4926376c-9c72-4d94-8505-959c770493a8-kube-api-access-cksfz\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593599 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-conf-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7e5339a-b073-4184-9acc-5a701276ea1c-agent-certs\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7e5339a-b073-4184-9acc-5a701276ea1c-konnectivity-ca\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-run\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.593847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593680 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-etc-selinux\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djclq\" (UniqueName: \"kubernetes.io/projected/edc57054-ce41-4e4e-97f6-b127bf151d23-kube-api-access-djclq\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593743 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-system-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593779 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22ea5ba-32e9-4552-802d-d11594d04f29-tmp-dir\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b7q6\" (UniqueName: \"kubernetes.io/projected/a22ea5ba-32e9-4552-802d-d11594d04f29-kube-api-access-7b7q6\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-slash\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593854 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-kubelet\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-var-lib-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.593980 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-lib-modules\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovn-node-metrics-cert\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-netns\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-daemon-config\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.594585 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594158 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-conf-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-registration-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-var-lib-kubelet\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594231 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blnvk\" (UniqueName: \"kubernetes.io/projected/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-kube-api-access-blnvk\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594261 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edc57054-ce41-4e4e-97f6-b127bf151d23-serviceca\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594299 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-etc-kubernetes\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594355 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-bin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594395 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-multus\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594451 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-hostroot\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594478 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594493 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-sys-fs\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594671 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594935 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7e5339a-b073-4184-9acc-5a701276ea1c-konnectivity-ca\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594977 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-multus-certs\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a22ea5ba-32e9-4552-802d-d11594d04f29-hosts-file\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cnibin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595058 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595068 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-etc-selinux\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.595401 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.594939 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cni-binary-copy\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595083 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-socket-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595127 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-netns\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595132 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-device-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595165 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-socket-dir-parent\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595182 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc57054-ce41-4e4e-97f6-b127bf151d23-host\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595210 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-system-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595330 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-k8s-cni-cncf-io\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595391 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-registration-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22ea5ba-32e9-4552-802d-d11594d04f29-tmp-dir\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.596207 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.595686 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-daemon-config\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.598947 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.598925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7e5339a-b073-4184-9acc-5a701276ea1c-agent-certs\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:03.603595 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.603567 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgr87\" (UniqueName: \"kubernetes.io/projected/06506c1c-88b0-42e1-ab42-8cc842decd18-kube-api-access-rgr87\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.604349 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.604327 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djclq\" (UniqueName: \"kubernetes.io/projected/edc57054-ce41-4e4e-97f6-b127bf151d23-kube-api-access-djclq\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.604588 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.604567 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b7q6\" (UniqueName: \"kubernetes.io/projected/a22ea5ba-32e9-4552-802d-d11594d04f29-kube-api-access-7b7q6\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.605065 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.605047 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd7gp\" (UniqueName: \"kubernetes.io/projected/e8a2984b-e706-43f8-bd87-dd5bc809c36e-kube-api-access-dd7gp\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.695173 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695098 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-var-lib-kubelet\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695173 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnvk\" (UniqueName: \"kubernetes.io/projected/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-kube-api-access-blnvk\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695173 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-systemd-units\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-conf\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzcv\" (UniqueName: \"kubernetes.io/projected/ada3c550-b52f-4236-88db-a07ed2c6fdda-kube-api-access-6rzcv\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s446\" (UniqueName: \"kubernetes.io/projected/0a2d0642-db9c-4bd7-a804-51167fdd62fd-kube-api-access-4s446\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-systemd\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-sys\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695268 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-host\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-dbus\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-netns\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695314 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-cnibin\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.695362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-kubelet-config\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-netd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysconfig\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-config\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695419 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-ovn\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-bin\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-script-lib\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-modprobe-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-etc-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695519 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ada3c550-b52f-4236-88db-a07ed2c6fdda-iptables-alerter-script\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695541 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-log-socket\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-kubelet\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-systemd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-node-log\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-kubernetes\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.695696 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-tuned\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada3c550-b52f-4236-88db-a07ed2c6fdda-host-slash\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-system-cni-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695690 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkgh\" (UniqueName: \"kubernetes.io/projected/4a228ff8-0200-4b6f-8948-9e351926a3b7-kube-api-access-vwkgh\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695720 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-tmp\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-env-overrides\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksfz\" (UniqueName: \"kubernetes.io/projected/4926376c-9c72-4d94-8505-959c770493a8-kube-api-access-cksfz\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-run\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695838 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-ovn\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-node-log\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-dbus\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695890 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-netd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.695909 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695928 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-etc-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.696468 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695948 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.695968 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret podName:81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:04.195950241 +0000 UTC m=+3.144768162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret") pod "global-pull-secret-syncer-2g5v9" (UID: "81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695969 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-cnibin\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-slash\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696007 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-modprobe-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-var-lib-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-lib-modules\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696115 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovn-node-metrics-cert\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696318 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-systemd\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-var-lib-kubelet\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-systemd-units\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696742 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-conf\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-log-socket\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-kubernetes\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697178 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.695923 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-kubelet-config\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696859 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-systemd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696904 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697206 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-sys\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697276 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-system-cni-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697296 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-slash\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697321 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada3c550-b52f-4236-88db-a07ed2c6fdda-host-slash\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697377 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.697399 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697418 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-lib-modules\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697422 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-host\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697245 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-run\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696803 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-kubelet\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-bin\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.697534 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs podName:0a2d0642-db9c-4bd7-a804-51167fdd62fd nodeName:}" failed. No retries permitted until 2026-11-01 15:47:04.197518788 +0000 UTC m=+3.146336693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs") pod "network-metrics-daemon-bdz9j" (UID: "0a2d0642-db9c-4bd7-a804-51167fdd62fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:03.697886 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697527 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-netns\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.696058 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysconfig\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697576 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-var-lib-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ada3c550-b52f-4236-88db-a07ed2c6fdda-iptables-alerter-script\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697891 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697916 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-config\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.697978 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-env-overrides\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.698073 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-script-lib\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.698430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.698370 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.700362 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.700316 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovn-node-metrics-cert\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.700475 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.700381 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-tmp\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.701249 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.701209 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-tuned\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.705741 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.705723 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:03.705854 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.705760 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:03.705854 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.705774 2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzdnx for pod openshift-network-diagnostics/network-check-target-2xm97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:03.705854 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:03.705825 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx podName:36ef09c4-10c5-4507-98c8-c94ec38f46f7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:04.205809301 +0000 UTC m=+3.154627212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nzdnx" (UniqueName: "kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx") pod "network-check-target-2xm97" (UID: "36ef09c4-10c5-4507-98c8-c94ec38f46f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:03.710117 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.710094 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzcv\" (UniqueName: \"kubernetes.io/projected/ada3c550-b52f-4236-88db-a07ed2c6fdda-kube-api-access-6rzcv\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.710916 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.710898 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnvk\" (UniqueName: \"kubernetes.io/projected/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-kube-api-access-blnvk\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.711088 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.711063 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s446\" (UniqueName: \"kubernetes.io/projected/0a2d0642-db9c-4bd7-a804-51167fdd62fd-kube-api-access-4s446\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:03.711497 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.711475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkgh\" (UniqueName: \"kubernetes.io/projected/4a228ff8-0200-4b6f-8948-9e351926a3b7-kube-api-access-vwkgh\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.712306 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.712280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksfz\" (UniqueName: \"kubernetes.io/projected/4926376c-9c72-4d94-8505-959c770493a8-kube-api-access-cksfz\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.793210 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.793172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:03.798069 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.798046 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:03.807642 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.807622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:03.812210 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:03.812187 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e5339a_b073_4184_9acc_5a701276ea1c.slice/crio-9388c520390b9b575fccf68a2923ba58f60cce989562911521a12f5131f97c4a WatchSource:0}: Error finding container 9388c520390b9b575fccf68a2923ba58f60cce989562911521a12f5131f97c4a: Status 404 returned error can't find the container with id 9388c520390b9b575fccf68a2923ba58f60cce989562911521a12f5131f97c4a Nov 01 15:47:03.813616 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.813597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:03.818189 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:03.818168 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22ea5ba_32e9_4552_802d_d11594d04f29.slice/crio-d97ad3dd117cbd1d90647fcd150f0dd4cb81c1c8d851d9dc54852f7fd840b0d5 WatchSource:0}: Error finding container d97ad3dd117cbd1d90647fcd150f0dd4cb81c1c8d851d9dc54852f7fd840b0d5: Status 404 returned error can't find the container with id d97ad3dd117cbd1d90647fcd150f0dd4cb81c1c8d851d9dc54852f7fd840b0d5 Nov 01 15:47:03.821925 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.821889 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b6ttz" Nov 01 15:47:03.824395 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:03.824369 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc57054_ce41_4e4e_97f6_b127bf151d23.slice/crio-6c44377411aba96da3f9ee0bdd884bcd7bed5bfbed5bb5b593c533321525218b WatchSource:0}: Error finding container 6c44377411aba96da3f9ee0bdd884bcd7bed5bfbed5bb5b593c533321525218b: Status 404 returned error can't find the container with id 6c44377411aba96da3f9ee0bdd884bcd7bed5bfbed5bb5b593c533321525218b Nov 01 15:47:03.830869 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.830847 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:03.834348 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:03.834319 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a2984b_e706_43f8_bd87_dd5bc809c36e.slice/crio-9124c1b00ef6d2bc43afdb2d4033b9d6dcaf936bb18c4c930ca4dc5bcb2ffaad WatchSource:0}: Error finding container 9124c1b00ef6d2bc43afdb2d4033b9d6dcaf936bb18c4c930ca4dc5bcb2ffaad: Status 404 returned error can't find the container with id 9124c1b00ef6d2bc43afdb2d4033b9d6dcaf936bb18c4c930ca4dc5bcb2ffaad Nov 01 15:47:03.842678 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:03.842654 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada3c550_b52f_4236_88db_a07ed2c6fdda.slice/crio-6121d84167620035b567b29c921b42e0a1fe83d4a7ca7fa8abd028a817fc4275 WatchSource:0}: Error finding container 6121d84167620035b567b29c921b42e0a1fe83d4a7ca7fa8abd028a817fc4275: Status 404 returned error can't find the container with id 6121d84167620035b567b29c921b42e0a1fe83d4a7ca7fa8abd028a817fc4275 Nov 01 15:47:03.858491 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.858468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:03.860617 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.860597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:03.870073 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:03.870049 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a228ff8_0200_4b6f_8948_9e351926a3b7.slice/crio-5a0675364f6c2fe46751d9b52f2d55c85ecacb060b93fac6a8fea44471d46edc WatchSource:0}: Error finding container 5a0675364f6c2fe46751d9b52f2d55c85ecacb060b93fac6a8fea44471d46edc: Status 404 returned error can't find the container with id 5a0675364f6c2fe46751d9b52f2d55c85ecacb060b93fac6a8fea44471d46edc Nov 01 15:47:03.870960 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:03.870785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:03.873317 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:03.873216 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0563d8c3_dfee_41e5_b9ed_81a71a05c8f7.slice/crio-149bd9b065f35a3577cef1c2b25852e87438e8f2d90feafed7bc489dafe2c08c WatchSource:0}: Error finding container 149bd9b065f35a3577cef1c2b25852e87438e8f2d90feafed7bc489dafe2c08c: Status 404 returned error can't find the container with id 149bd9b065f35a3577cef1c2b25852e87438e8f2d90feafed7bc489dafe2c08c Nov 01 15:47:03.881311 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:03.881288 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4926376c_9c72_4d94_8505_959c770493a8.slice/crio-f9485f27d362087ade8dd7c28848a97cdbeae94df836cdb8870ba259cf176a39 WatchSource:0}: Error finding container f9485f27d362087ade8dd7c28848a97cdbeae94df836cdb8870ba259cf176a39: Status 404 returned error can't find the container with id f9485f27d362087ade8dd7c28848a97cdbeae94df836cdb8870ba259cf176a39 Nov 01 15:47:04.200476 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.200441 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:04.200646 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.200488 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:04.200718 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.200635 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:04.200718 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.200716 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs podName:0a2d0642-db9c-4bd7-a804-51167fdd62fd nodeName:}" failed. No retries permitted until 2026-11-01 15:47:05.200697882 +0000 UTC m=+4.149515802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs") pod "network-metrics-daemon-bdz9j" (UID: "0a2d0642-db9c-4bd7-a804-51167fdd62fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:04.200891 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.200641 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:04.200891 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.200806 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret podName:81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:05.200789736 +0000 UTC m=+4.149607654 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret") pod "global-pull-secret-syncer-2g5v9" (UID: "81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:04.301784 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.301732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:04.301941 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.301895 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:04.301941 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.301911 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:04.301941 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.301920 2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzdnx for pod openshift-network-diagnostics/network-check-target-2xm97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:04.302125 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.301976 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx podName:36ef09c4-10c5-4507-98c8-c94ec38f46f7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:05.301958153 +0000 UTC m=+4.250776073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzdnx" (UniqueName: "kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx") pod "network-check-target-2xm97" (UID: "36ef09c4-10c5-4507-98c8-c94ec38f46f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:04.519743 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.519571 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:03 +0000 UTC" deadline="2028-06-05 15:06:01.838023789 +0000 UTC" Nov 01 15:47:04.519743 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.519639 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13967h18m57.318388551s" Nov 01 15:47:04.557211 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.556723 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:04.557211 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:04.556857 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:04.568995 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.568889 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" event={"ID":"06506c1c-88b0-42e1-ab42-8cc842decd18","Type":"ContainerStarted","Data":"c7273e3bd74ed6a5a2ba8093cba50f15427e53bf06692650840370edefaff1ce"} Nov 01 15:47:04.578459 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.578389 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" event={"ID":"4926376c-9c72-4d94-8505-959c770493a8","Type":"ContainerStarted","Data":"f9485f27d362087ade8dd7c28848a97cdbeae94df836cdb8870ba259cf176a39"} Nov 01 15:47:04.582010 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.581978 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p66vf" event={"ID":"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7","Type":"ContainerStarted","Data":"149bd9b065f35a3577cef1c2b25852e87438e8f2d90feafed7bc489dafe2c08c"} Nov 01 15:47:04.590622 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.590589 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q52bf" event={"ID":"ada3c550-b52f-4236-88db-a07ed2c6fdda","Type":"ContainerStarted","Data":"6121d84167620035b567b29c921b42e0a1fe83d4a7ca7fa8abd028a817fc4275"} Nov 01 15:47:04.595434 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.595339 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8fxdr" event={"ID":"a22ea5ba-32e9-4552-802d-d11594d04f29","Type":"ContainerStarted","Data":"d97ad3dd117cbd1d90647fcd150f0dd4cb81c1c8d851d9dc54852f7fd840b0d5"} Nov 01 15:47:04.602021 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.601929 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"5a0675364f6c2fe46751d9b52f2d55c85ecacb060b93fac6a8fea44471d46edc"} Nov 01 15:47:04.613350 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.613316 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b6ttz" event={"ID":"e8a2984b-e706-43f8-bd87-dd5bc809c36e","Type":"ContainerStarted","Data":"9124c1b00ef6d2bc43afdb2d4033b9d6dcaf936bb18c4c930ca4dc5bcb2ffaad"} Nov 01 15:47:04.615186 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.615159 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rsjf4" event={"ID":"edc57054-ce41-4e4e-97f6-b127bf151d23","Type":"ContainerStarted","Data":"6c44377411aba96da3f9ee0bdd884bcd7bed5bfbed5bb5b593c533321525218b"} Nov 01 15:47:04.618490 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:04.618461 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cvct9" event={"ID":"f7e5339a-b073-4184-9acc-5a701276ea1c","Type":"ContainerStarted","Data":"9388c520390b9b575fccf68a2923ba58f60cce989562911521a12f5131f97c4a"} Nov 01 15:47:05.211159 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.209805 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:05.211159 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.210092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:05.211159 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.210231 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:05.211159 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.210292 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret podName:81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:07.210273721 +0000 UTC m=+6.159091639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret") pod "global-pull-secret-syncer-2g5v9" (UID: "81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:05.211159 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.210668 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:05.211159 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.210718 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs podName:0a2d0642-db9c-4bd7-a804-51167fdd62fd nodeName:}" failed. No retries permitted until 2026-11-01 15:47:07.21070192 +0000 UTC m=+6.159519825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs") pod "network-metrics-daemon-bdz9j" (UID: "0a2d0642-db9c-4bd7-a804-51167fdd62fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:05.310780 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.310619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:05.310941 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.310784 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:05.310941 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.310808 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:05.310941 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.310822 2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzdnx for pod openshift-network-diagnostics/network-check-target-2xm97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:05.310941 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.310881 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx podName:36ef09c4-10c5-4507-98c8-c94ec38f46f7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:07.310861678 +0000 UTC m=+6.259679598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzdnx" (UniqueName: "kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx") pod "network-check-target-2xm97" (UID: "36ef09c4-10c5-4507-98c8-c94ec38f46f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:05.520592 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.520539 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:03 +0000 UTC" deadline="2028-04-30 17:39:09.259770466 +0000 UTC" Nov 01 15:47:05.520592 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.520578 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13105h52m3.739196745s" Nov 01 15:47:05.560923 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.560860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:05.561080 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.560992 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:05.561428 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.561408 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:05.561542 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:05.561523 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:05.629553 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.629508 2579 generic.go:356] "Generic (PLEG): container finished" podID="92683d17f4ef39773bb8bc1b1459025a" containerID="4ad8722323c7e02cec9187625b11a5654be046a5cba624f55e7d198548360c5c" exitCode=0 Nov 01 15:47:05.629710 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.629592 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerDied","Data":"4ad8722323c7e02cec9187625b11a5654be046a5cba624f55e7d198548360c5c"} Nov 01 15:47:05.640605 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:05.640179 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"6555262b15c85369c5c728ca31cfdef1","Type":"ContainerStarted","Data":"b31a3718a182f952a7ba978d8e68111da6907fce31f064325591a86bc25694d1"} Nov 01 15:47:06.556303 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:06.556276 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:06.556830 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:06.556394 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:06.668934 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:06.668899 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"63ae19f081e708ec74a910bf3c54cd46fe5e7ac891d89ac659aa92baf3c221e9"} Nov 01 15:47:06.683898 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:06.683845 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" podStartSLOduration=4.683827278 podStartE2EDuration="4.683827278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:05.657953313 +0000 UTC m=+4.606771238" watchObservedRunningTime="2026-11-01 15:47:06.683827278 +0000 UTC m=+5.632645205" Nov 01 15:47:07.234291 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:07.234209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:07.234291 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:07.234283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:07.234549 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.234520 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:07.234641 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.234619 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret podName:81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:11.234598817 +0000 UTC m=+10.183416743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret") pod "global-pull-secret-syncer-2g5v9" (UID: "81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:07.235058 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.235036 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:07.235168 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.235091 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs podName:0a2d0642-db9c-4bd7-a804-51167fdd62fd nodeName:}" failed. No retries permitted until 2026-11-01 15:47:11.235076142 +0000 UTC m=+10.183894063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs") pod "network-metrics-daemon-bdz9j" (UID: "0a2d0642-db9c-4bd7-a804-51167fdd62fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:07.335398 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:07.335360 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:07.335574 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.335518 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:07.335574 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.335537 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:07.335574 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.335550 2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzdnx for pod openshift-network-diagnostics/network-check-target-2xm97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:07.335732 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.335605 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx podName:36ef09c4-10c5-4507-98c8-c94ec38f46f7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:11.335587428 +0000 UTC m=+10.284405349 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzdnx" (UniqueName: "kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx") pod "network-check-target-2xm97" (UID: "36ef09c4-10c5-4507-98c8-c94ec38f46f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:07.558675 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:07.558608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:07.559119 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.558707 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:07.559119 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:07.558918 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:07.559119 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:07.559016 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:07.678233 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:07.677573 2579 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="e6dcb4972966df4a0425cb6458b7f367b807d4d3f4b4b6f51267ad0c3754f391" exitCode=0 Nov 01 15:47:07.678233 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:07.677698 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" event={"ID":"4926376c-9c72-4d94-8505-959c770493a8","Type":"ContainerDied","Data":"e6dcb4972966df4a0425cb6458b7f367b807d4d3f4b4b6f51267ad0c3754f391"} Nov 01 15:47:07.698541 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:07.698490 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" podStartSLOduration=5.698472261 podStartE2EDuration="5.698472261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:06.684268421 +0000 UTC m=+5.633086348" watchObservedRunningTime="2026-11-01 15:47:07.698472261 +0000 UTC m=+6.647290190" Nov 01 15:47:08.556718 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:08.556146 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:08.556718 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:08.556371 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:08.683569 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:08.683519 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cvct9" event={"ID":"f7e5339a-b073-4184-9acc-5a701276ea1c","Type":"ContainerStarted","Data":"e587b424333db55404874329b1b2e94f96bd6b5332911b673ce64e9b41e2b0c5"} Nov 01 15:47:09.558676 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:09.558607 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:09.558829 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:09.558741 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:09.559147 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:09.559129 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:09.559256 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:09.559235 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:09.690452 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:09.690407 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rsjf4" event={"ID":"edc57054-ce41-4e4e-97f6-b127bf151d23","Type":"ContainerStarted","Data":"5588ed7f674ea0721236ec95701c6c42f0d793dd590dc6a27d7e22fc688483ae"} Nov 01 15:47:09.694903 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:09.694835 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" event={"ID":"06506c1c-88b0-42e1-ab42-8cc842decd18","Type":"ContainerStarted","Data":"7966319c0519a8641ed1412b6fd1be31675b768f2e0acfc2219aaa83140cf2e7"} Nov 01 15:47:09.705413 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:09.705369 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cvct9" podStartSLOduration=4.894065802 podStartE2EDuration="8.705352266s" totalImagesPullingTime="3.811286464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:08.698317312 +0000 UTC m=+7.647135239" watchObservedRunningTime="2026-11-01 15:47:09.705352266 +0000 UTC m=+8.654170195" Nov 01 15:47:09.705835 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:09.705798 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rsjf4" podStartSLOduration=3.6947677 podStartE2EDuration="8.70578916s" totalImagesPullingTime="5.01102146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:09.704875814 +0000 UTC m=+8.653693741" watchObservedRunningTime="2026-11-01 15:47:09.70578916 +0000 UTC m=+8.654607090" Nov 01 15:47:10.556348 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:10.555863 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:10.556348 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:10.555972 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:10.926213 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:10.926180 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 15:47:11.278314 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.277607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:11.278314 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.277670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:11.278314 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.277803 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:11.278314 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.277869 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret podName:81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:19.277850437 +0000 UTC m=+18.226668359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret") pod "global-pull-secret-syncer-2g5v9" (UID: "81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:11.278314 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.277871 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:11.278314 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.277907 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs podName:0a2d0642-db9c-4bd7-a804-51167fdd62fd nodeName:}" failed. No retries permitted until 2026-11-01 15:47:19.277897719 +0000 UTC m=+18.226715630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs") pod "network-metrics-daemon-bdz9j" (UID: "0a2d0642-db9c-4bd7-a804-51167fdd62fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:11.379130 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.378592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:11.379130 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.378722 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:11.379130 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.378734 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:11.379130 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.378744 2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzdnx for pod openshift-network-diagnostics/network-check-target-2xm97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:11.379130 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.378805 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx podName:36ef09c4-10c5-4507-98c8-c94ec38f46f7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:19.378788415 +0000 UTC m=+18.327606332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzdnx" (UniqueName: "kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx") pod "network-check-target-2xm97" (UID: "36ef09c4-10c5-4507-98c8-c94ec38f46f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:11.557578 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.557554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:11.557682 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.557636 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:11.557802 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.557782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:11.557922 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:11.557901 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:11.687570 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.687433 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T15:47:10.92620647Z","UUID":"93c5ee10-6239-492c-956b-7909f0a606d5","Handler":null,"Name":"","Endpoint":""} Nov 01 15:47:11.689332 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.689302 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 15:47:11.689455 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.689346 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 15:47:11.703057 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.702343 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8fxdr" event={"ID":"a22ea5ba-32e9-4552-802d-d11594d04f29","Type":"ContainerStarted","Data":"a861dccf94cb9b99fd2a22c927035ec3fe148fc123e2c9396544523cd0b6170f"} Nov 01 15:47:11.712309 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.711689 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b6ttz" event={"ID":"e8a2984b-e706-43f8-bd87-dd5bc809c36e","Type":"ContainerStarted","Data":"5be5d76903f541e73342ca0d223372ce60ac09cef38b4f7efecbfaba8aa4e24b"} Nov 01 15:47:11.715206 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.715129 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" event={"ID":"06506c1c-88b0-42e1-ab42-8cc842decd18","Type":"ContainerStarted","Data":"b3c5d9a745b440264966ded87af4792e5f710b30084de9829e29679817bad183"} Nov 01 15:47:11.721765 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:11.720480 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8fxdr" podStartSLOduration=3.225732224 podStartE2EDuration="10.720467131s" totalImagesPullingTime="7.494734907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:11.720069082 +0000 UTC m=+10.668887024" watchObservedRunningTime="2026-11-01 15:47:11.720467131 +0000 UTC m=+10.669285058" Nov 01 15:47:12.556814 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:12.556559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:12.557262 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:12.556915 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:12.719731 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:12.719691 2579 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="9a8ed9e695a18ade97eae406710a69e488832bdd25f7e1d1ca10c920e8b35430" exitCode=0 Nov 01 15:47:12.719929 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:12.719806 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" event={"ID":"4926376c-9c72-4d94-8505-959c770493a8","Type":"ContainerDied","Data":"9a8ed9e695a18ade97eae406710a69e488832bdd25f7e1d1ca10c920e8b35430"} Nov 01 15:47:12.723275 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:12.723253 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" event={"ID":"06506c1c-88b0-42e1-ab42-8cc842decd18","Type":"ContainerStarted","Data":"e88ca5a76aa6b0ad9260a3ab271ef6f961059b3f650a564eb5d94649d12d9509"} Nov 01 15:47:12.740344 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:12.740264 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-b6ttz" podStartSLOduration=4.025752251 podStartE2EDuration="11.740246397s" totalImagesPullingTime="7.714494146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:11.740801614 +0000 UTC m=+10.689619735" watchObservedRunningTime="2026-11-01 15:47:12.740246397 +0000 UTC m=+11.689064323" Nov 01 15:47:13.556868 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.556834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:13.557246 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:13.556959 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:13.557395 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.557377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:13.557471 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:13.557457 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:13.650461 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.650426 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:13.651219 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.651194 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:13.664847 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.664807 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" podStartSLOduration=4.77529147 podStartE2EDuration="12.664795322s" totalImagesPullingTime="7.889503852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:12.75595525 +0000 UTC m=+11.704773178" watchObservedRunningTime="2026-11-01 15:47:13.664795322 +0000 UTC m=+12.613613283" Nov 01 15:47:13.727305 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.727220 2579 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="c4f340dc9fdd41056c9ffe19a9d8be062d61d66e4ef05b7452819293d63fdd0d" exitCode=0 Nov 01 15:47:13.727457 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.727315 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" event={"ID":"4926376c-9c72-4d94-8505-959c770493a8","Type":"ContainerDied","Data":"c4f340dc9fdd41056c9ffe19a9d8be062d61d66e4ef05b7452819293d63fdd0d"} Nov 01 15:47:13.729337 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.729277 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q52bf" event={"ID":"ada3c550-b52f-4236-88db-a07ed2c6fdda","Type":"ContainerStarted","Data":"0e9664257aabf434281afdf0897e5fe0d72eb989444fcdf452f771523612cabe"} Nov 01 15:47:13.759127 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:13.759075 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q52bf" podStartSLOduration=4.860644288 podStartE2EDuration="12.75905885s" totalImagesPullingTime="7.898414562s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:13.758728036 +0000 UTC m=+12.707545962" watchObservedRunningTime="2026-11-01 15:47:13.75905885 +0000 UTC m=+12.707876777" Nov 01 15:47:14.556507 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:14.556482 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:14.556617 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:14.556597 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:14.625817 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:14.625787 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:14.626741 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:14.626720 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:14.734091 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:14.734006 2579 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="2fe0756340012a74b2f9fbe77da8206e49c0c316ed235f39f796f98cf6925ef3" exitCode=0 Nov 01 15:47:14.734233 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:14.734095 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" event={"ID":"4926376c-9c72-4d94-8505-959c770493a8","Type":"ContainerDied","Data":"2fe0756340012a74b2f9fbe77da8206e49c0c316ed235f39f796f98cf6925ef3"} Nov 01 15:47:14.736138 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:14.736115 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p66vf" event={"ID":"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7","Type":"ContainerStarted","Data":"f78b3319641f11f2aef65ccfef922d064a901b97f48246d35ddde26187cdf373"} Nov 01 15:47:14.771360 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:14.771309 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p66vf" podStartSLOduration=3.651915522 podStartE2EDuration="13.771293024s" totalImagesPullingTime="10.119377502s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:14.770808281 +0000 UTC m=+13.719626209" watchObservedRunningTime="2026-11-01 15:47:14.771293024 +0000 UTC m=+13.720110953" Nov 01 15:47:15.556893 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:15.556857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:15.557081 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:15.556990 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:15.557081 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:15.557050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:15.557269 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:15.557247 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:16.556039 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:16.556007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:16.556610 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:16.556122 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:17.556495 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:17.556299 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:17.556972 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:17.556314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:17.556972 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:17.556611 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:17.556972 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:17.556671 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:17.747219 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:17.747187 2579 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="464c956c6a1bfd4c44a6544ef0c1f4f67b9292a462b6a2890bb9393f8aa2b185" exitCode=0 Nov 01 15:47:17.747357 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:17.747237 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" event={"ID":"4926376c-9c72-4d94-8505-959c770493a8","Type":"ContainerDied","Data":"464c956c6a1bfd4c44a6544ef0c1f4f67b9292a462b6a2890bb9393f8aa2b185"} Nov 01 15:47:18.556430 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.556275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:18.556601 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:18.556495 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:18.752531 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.752450 2579 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="6c470e973f02041a37c73c286f974933f328f1b7bd376ce051fc9625125ee49f" exitCode=0 Nov 01 15:47:18.752668 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.752534 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" event={"ID":"4926376c-9c72-4d94-8505-959c770493a8","Type":"ContainerDied","Data":"6c470e973f02041a37c73c286f974933f328f1b7bd376ce051fc9625125ee49f"} Nov 01 15:47:18.755344 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.755328 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:47:18.755719 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.755699 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a228ff8-0200-4b6f-8948-9e351926a3b7" containerID="3555c1050014611b1d2fc2394bc085dabcf031d243fb0a937975dcba6eac6a7d" exitCode=1 Nov 01 15:47:18.755802 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.755741 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"82284db94ac9fdb7170b8135c528f3fbb300e1b133d5c19eeb21e86325b2368c"} Nov 01 15:47:18.755839 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.755799 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"5651f94191ef92aa539fc1cd51ed4181c20d216e3dd191919a1a6361e0bbe192"} Nov 01 15:47:18.755839 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.755831 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"1a150d5a43757c7d61fae48ffaa93f512ed60b9e4cd15626f9631dee1b6387cf"} Nov 01 15:47:18.755908 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.755863 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"aa83e375d5c4a3aecd293112eec204d260e8c034b1f226e4d4a5bc42e89553bc"} Nov 01 15:47:18.755908 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.755891 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerDied","Data":"3555c1050014611b1d2fc2394bc085dabcf031d243fb0a937975dcba6eac6a7d"} Nov 01 15:47:18.755970 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:18.755926 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"d0522a7e7d226b39b51681b48c0d7cfe731fd1a677c85a5dd1bed43ab415bf4f"} Nov 01 15:47:19.338746 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:19.338712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:19.338927 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:19.338781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:19.338927 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.338891 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:19.338993 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.338960 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret podName:81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:35.338945809 +0000 UTC m=+34.287763728 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret") pod "global-pull-secret-syncer-2g5v9" (UID: "81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1") : object "kube-system"/"original-pull-secret" not registered Nov 01 15:47:19.338993 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.338891 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:19.339063 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.339022 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs podName:0a2d0642-db9c-4bd7-a804-51167fdd62fd nodeName:}" failed. No retries permitted until 2026-11-01 15:47:35.339010136 +0000 UTC m=+34.287828050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs") pod "network-metrics-daemon-bdz9j" (UID: "0a2d0642-db9c-4bd7-a804-51167fdd62fd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 15:47:19.439864 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:19.439827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:19.440031 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.439942 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 15:47:19.440031 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.439956 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 15:47:19.440031 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.439965 2579 projected.go:196] Error preparing data for projected volume kube-api-access-nzdnx for pod openshift-network-diagnostics/network-check-target-2xm97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:19.440031 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.440012 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx podName:36ef09c4-10c5-4507-98c8-c94ec38f46f7 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:35.440000832 +0000 UTC m=+34.388818736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nzdnx" (UniqueName: "kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx") pod "network-check-target-2xm97" (UID: "36ef09c4-10c5-4507-98c8-c94ec38f46f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 15:47:19.556547 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:19.556516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:19.556710 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.556637 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:19.556710 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:19.556695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:19.557033 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:19.556806 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:19.761078 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:19.760996 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" event={"ID":"4926376c-9c72-4d94-8505-959c770493a8","Type":"ContainerStarted","Data":"7fc18772f89cbbc7b25fc3bcefc528875594024e1e5ce3beb5155c87c872dae8"} Nov 01 15:47:19.783505 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:19.783449 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bmsrc" podStartSLOduration=8.307547491 podStartE2EDuration="18.783434626s" totalImagesPullingTime="10.475887135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:19.782890362 +0000 UTC m=+18.731708293" watchObservedRunningTime="2026-11-01 15:47:19.783434626 +0000 UTC m=+18.732252552" Nov 01 15:47:20.556525 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:20.556496 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:20.556658 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:20.556594 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:20.765787 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:20.765687 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:47:20.766264 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:20.766072 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"58a03905c288d55d5fb17a3b11593319339ff03fdcab920b466a9c2026281c08"} Nov 01 15:47:21.556571 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:21.556538 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:21.556779 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:21.556642 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:21.556779 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:21.556698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:21.556893 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:21.556800 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:22.556424 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:22.556406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:22.556931 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:22.556505 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:22.772986 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:22.772921 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:47:22.773269 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:22.773246 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"734b8fe9ed91f37b5fe1f7aef8df16868fe43cd0cdf5b59272e395797f3f1eb2"} Nov 01 15:47:22.773538 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:22.773522 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:22.773644 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:22.773622 2579 scope.go:122] "RemoveContainer" containerID="3555c1050014611b1d2fc2394bc085dabcf031d243fb0a937975dcba6eac6a7d" Nov 01 15:47:22.789425 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:22.789304 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:23.556817 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:23.556784 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:23.557275 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:23.556889 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:23.557275 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:23.556940 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:23.557275 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:23.557055 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:23.779628 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:23.779594 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:47:23.780025 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:23.780002 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" event={"ID":"4a228ff8-0200-4b6f-8948-9e351926a3b7","Type":"ContainerStarted","Data":"ceb987d469d1bf34eb8a88ffa3d76bad5018d29335ab98ffb3de7a4f23f08dc5"} Nov 01 15:47:23.780347 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:23.780327 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:23.780440 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:23.780398 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:23.802777 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:23.799703 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:23.807811 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:23.806613 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" podStartSLOduration=8.678881301 podStartE2EDuration="22.806595014s" totalImagesPullingTime="14.127713713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:23.804990818 +0000 UTC m=+22.753808750" watchObservedRunningTime="2026-11-01 15:47:23.806595014 +0000 UTC m=+22.755412941" Nov 01 15:47:24.257663 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:24.257594 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2g5v9"] Nov 01 15:47:24.257875 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:24.257702 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:24.257875 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:24.257834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:24.260684 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:24.260655 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2xm97"] Nov 01 15:47:24.260842 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:24.260777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:24.260905 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:24.260858 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:24.261389 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:24.261363 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdz9j"] Nov 01 15:47:24.261516 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:24.261495 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:24.261633 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:24.261604 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:26.555969 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:26.555937 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:26.556448 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:26.555995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:26.556448 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:26.556086 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:26.556448 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:26.556093 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:26.556448 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:26.556187 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:26.556448 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:26.556274 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:28.556037 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:28.555946 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:28.556037 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:28.555996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:28.556421 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:28.555970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:28.556421 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:28.556084 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bdz9j" podUID="0a2d0642-db9c-4bd7-a804-51167fdd62fd" Nov 01 15:47:28.556421 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:28.556125 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2xm97" podUID="36ef09c4-10c5-4507-98c8-c94ec38f46f7" Nov 01 15:47:28.556421 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:28.556194 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2g5v9" podUID="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" Nov 01 15:47:28.969367 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:28.969292 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:29.091793 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:29.091744 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:30.302876 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.302797 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeReady" Nov 01 15:47:30.303221 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.302914 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 15:47:30.342867 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.342833 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-b7q5z"] Nov 01 15:47:30.345433 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.345413 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:30.353266 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.353247 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gx6wg"] Nov 01 15:47:30.354441 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.353956 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 15:47:30.354441 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.354080 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-srrqq\"" Nov 01 15:47:30.354441 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.354177 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 15:47:30.354894 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.354872 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 15:47:30.356132 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.356116 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7q5z"] Nov 01 15:47:30.356226 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.356189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.365733 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.365710 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gx6wg"] Nov 01 15:47:30.456673 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.456648 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 15:47:30.456814 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.456674 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 15:47:30.456940 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.456927 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wl7pm\"" Nov 01 15:47:30.526742 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.526718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-cert\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:30.526848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.526776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370dbf42-367c-48ac-abb8-86c01bfe2b54-tmp-dir\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.526848 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.526808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/370dbf42-367c-48ac-abb8-86c01bfe2b54-config-volume\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.526925 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.526880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.526925 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.526904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nrjm\" (UniqueName: \"kubernetes.io/projected/370dbf42-367c-48ac-abb8-86c01bfe2b54-kube-api-access-9nrjm\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.526925 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.526919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brr4h\" (UniqueName: \"kubernetes.io/projected/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-kube-api-access-brr4h\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:30.556140 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.556062 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:30.556140 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.556085 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:30.556329 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.556236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:30.561887 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.561860 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 15:47:30.561887 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.561872 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 15:47:30.561887 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.561858 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2bqxm\"" Nov 01 15:47:30.562200 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.562121 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 15:47:30.562200 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.562147 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4p8cp\"" Nov 01 15:47:30.562200 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.562151 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 15:47:30.562200 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.562175 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qm9xj\"" Nov 01 15:47:30.627441 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.627420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370dbf42-367c-48ac-abb8-86c01bfe2b54-tmp-dir\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.627520 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.627448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/370dbf42-367c-48ac-abb8-86c01bfe2b54-config-volume\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.627520 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.627465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.627520 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.627487 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrjm\" (UniqueName: \"kubernetes.io/projected/370dbf42-367c-48ac-abb8-86c01bfe2b54-kube-api-access-9nrjm\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.627639 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:30.627586 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:30.627639 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:30.627636 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls podName:370dbf42-367c-48ac-abb8-86c01bfe2b54 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:31.127620169 +0000 UTC m=+30.076438073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls") pod "dns-default-gx6wg" (UID: "370dbf42-367c-48ac-abb8-86c01bfe2b54") : secret "dns-default-metrics-tls" not found Nov 01 15:47:30.627741 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.627688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brr4h\" (UniqueName: \"kubernetes.io/projected/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-kube-api-access-brr4h\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:30.627817 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.627791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-cert\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:30.628142 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.628123 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370dbf42-367c-48ac-abb8-86c01bfe2b54-tmp-dir\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.628270 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.628253 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/370dbf42-367c-48ac-abb8-86c01bfe2b54-config-volume\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.632000 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.631855 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-cert\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:30.637639 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.637613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrjm\" (UniqueName: \"kubernetes.io/projected/370dbf42-367c-48ac-abb8-86c01bfe2b54-kube-api-access-9nrjm\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:30.637717 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.637705 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brr4h\" (UniqueName: \"kubernetes.io/projected/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-kube-api-access-brr4h\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:30.657557 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.657540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:30.933857 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:30.933788 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-b7q5z"] Nov 01 15:47:30.937112 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:30.937087 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9e4381_b8a2_4d0d_ade3_75df4d07810f.slice/crio-4a32f04ab0b6239f10d269bfc5393003e4673900e8be00c3616073d1fce2c96d WatchSource:0}: Error finding container 4a32f04ab0b6239f10d269bfc5393003e4673900e8be00c3616073d1fce2c96d: Status 404 returned error can't find the container with id 4a32f04ab0b6239f10d269bfc5393003e4673900e8be00c3616073d1fce2c96d Nov 01 15:47:31.130960 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:31.130931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:31.131102 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:31.131070 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:31.131145 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:31.131127 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls podName:370dbf42-367c-48ac-abb8-86c01bfe2b54 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:32.131110568 +0000 UTC m=+31.079928488 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls") pod "dns-default-gx6wg" (UID: "370dbf42-367c-48ac-abb8-86c01bfe2b54") : secret "dns-default-metrics-tls" not found Nov 01 15:47:31.799539 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:31.799496 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7q5z" event={"ID":"8b9e4381-b8a2-4d0d-ade3-75df4d07810f","Type":"ContainerStarted","Data":"4a32f04ab0b6239f10d269bfc5393003e4673900e8be00c3616073d1fce2c96d"} Nov 01 15:47:31.979664 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:31.979627 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:32.053135 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:32.053055 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:32.138033 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:32.138002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:32.138196 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:32.138175 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:32.138267 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:32.138255 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls podName:370dbf42-367c-48ac-abb8-86c01bfe2b54 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:34.138235818 +0000 UTC m=+33.087053736 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls") pod "dns-default-gx6wg" (UID: "370dbf42-367c-48ac-abb8-86c01bfe2b54") : secret "dns-default-metrics-tls" not found Nov 01 15:47:32.333502 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:32.333428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:32.400800 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:32.400764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:33.822683 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:33.822647 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-b7q5z" event={"ID":"8b9e4381-b8a2-4d0d-ade3-75df4d07810f","Type":"ContainerStarted","Data":"16754f28204bd28644a9f23a00c342857c99856379d8e2a5b81268e0567402d7"} Nov 01 15:47:33.838399 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:33.838350 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-b7q5z" podStartSLOduration=1.082618811 podStartE2EDuration="3.838336822s" totalImagesPullingTime="2.755718011s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:33.837743671 +0000 UTC m=+32.786561597" watchObservedRunningTime="2026-11-01 15:47:33.838336822 +0000 UTC m=+32.787154748" Nov 01 15:47:34.153545 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:34.153515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:34.153672 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:34.153658 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:34.153733 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:34.153723 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls podName:370dbf42-367c-48ac-abb8-86c01bfe2b54 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:38.153708133 +0000 UTC m=+37.102526051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls") pod "dns-default-gx6wg" (UID: "370dbf42-367c-48ac-abb8-86c01bfe2b54") : secret "dns-default-metrics-tls" not found Nov 01 15:47:35.055245 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.055172 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-lsszq"] Nov 01 15:47:35.088479 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.088444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.121440 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.121417 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jv28d-6kkx9"] Nov 01 15:47:35.121571 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.121538 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.124528 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.124509 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lsszq"] Nov 01 15:47:35.124637 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.124546 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jv28d-6kkx9"] Nov 01 15:47:35.124677 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.124630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.131347 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.131316 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 15:47:35.131452 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.131361 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtfck\"" Nov 01 15:47:35.131452 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.131413 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jbzz7\"" Nov 01 15:47:35.157999 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.157976 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 15:47:35.158111 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.158004 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 15:47:35.158200 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.158135 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 15:47:35.158278 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.158249 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 15:47:35.162335 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.162317 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 15:47:35.193451 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.193427 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.261599 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da994e56-8f1d-4f00-893a-670984816adc-data-volume\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.261697 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261601 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.261697 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.261794 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.261835 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261802 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn6rl\" (UniqueName: \"kubernetes.io/projected/da994e56-8f1d-4f00-893a-670984816adc-kube-api-access-rn6rl\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.261835 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.261908 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.261955 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261926 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.262005 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.261988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da994e56-8f1d-4f00-893a-670984816adc-crio-socket\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.307425 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.307355 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6hrlt"] Nov 01 15:47:35.314049 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.314027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.337819 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.337790 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6hrlt"] Nov 01 15:47:35.337921 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.337856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.343195 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.343130 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 15:47:35.343373 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.343351 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 15:47:35.343463 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.343444 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rkntg\"" Nov 01 15:47:35.343535 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.343461 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 15:47:35.343651 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.343633 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 15:47:35.362829 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.362807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.362921 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.362838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.363010 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.362983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da994e56-8f1d-4f00-893a-670984816adc-crio-socket\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.363052 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da994e56-8f1d-4f00-893a-670984816adc-data-volume\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.363096 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363075 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.363144 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.363225 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363204 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da994e56-8f1d-4f00-893a-670984816adc-crio-socket\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.363279 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363229 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.363330 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:35.363380 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.363339 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:35.363428 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363337 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.363428 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.363396 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:35.863376132 +0000 UTC m=+34.812194052 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:35.363428 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.363395 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 15:47:35.363686 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da994e56-8f1d-4f00-893a-670984816adc-data-volume\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.363808 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.363791 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 01 15:47:35.363869 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6rl\" (UniqueName: \"kubernetes.io/projected/da994e56-8f1d-4f00-893a-670984816adc-kube-api-access-rn6rl\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.363869 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.363861 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs podName:0a2d0642-db9c-4bd7-a804-51167fdd62fd nodeName:}" failed. No retries permitted until 2026-11-01 15:48:07.36384206 +0000 UTC m=+66.312659979 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs") pod "network-metrics-daemon-bdz9j" (UID: "0a2d0642-db9c-4bd7-a804-51167fdd62fd") : secret "metrics-daemon-secret" not found Nov 01 15:47:35.364094 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.363923 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.364146 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.364125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:35.364939 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.364775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.365043 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.364952 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:35.86491975 +0000 UTC m=+34.813737673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : configmap "kube-rbac-proxy" not found Nov 01 15:47:35.365114 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.365081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.367940 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.367566 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:35.377133 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.377113 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6rl\" (UniqueName: \"kubernetes.io/projected/da994e56-8f1d-4f00-893a-670984816adc-kube-api-access-rn6rl\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.377361 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.377343 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.378084 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.378066 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:35.408032 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.408011 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.464812 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.464786 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:35.465054 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.465021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:35.465177 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.465077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-cabundle\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.465177 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.465124 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-key\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.465177 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.465153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4sb5\" (UniqueName: \"kubernetes.io/projected/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-kube-api-access-d4sb5\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.468418 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.468386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:35.551627 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.551601 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.555381 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.555356 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4"] Nov 01 15:47:35.566286 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.566227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-cabundle\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.566286 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.566270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-key\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.566438 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.566300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sb5\" (UniqueName: \"kubernetes.io/projected/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-kube-api-access-d4sb5\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.566862 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.566843 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-cabundle\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.568402 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.568383 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-key\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.573088 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.573069 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4"] Nov 01 15:47:35.573179 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.573163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" Nov 01 15:47:35.579841 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.579818 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:35.579932 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.579894 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4c2bg\"" Nov 01 15:47:35.580002 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.579982 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 15:47:35.582058 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.582038 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sb5\" (UniqueName: \"kubernetes.io/projected/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-kube-api-access-d4sb5\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.659878 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.659841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:35.666792 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.666765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvz7\" (UniqueName: \"kubernetes.io/projected/4bf6473d-8d38-41da-9d38-d71ddc4957ad-kube-api-access-9fvz7\") pod \"migrator-66df5d496b-fq2w4\" (UID: \"4bf6473d-8d38-41da-9d38-d71ddc4957ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" Nov 01 15:47:35.667329 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.667308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.672106 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.672084 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2g5v9"] Nov 01 15:47:35.673152 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.673136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:35.674881 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:35.674858 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dc536f_2ef1_43d3_aa56_ccbe4ecf66b1.slice/crio-73444df967788cd0a0b71d484a2ae265da3589ad4c2e3542e0fd7e7a63e0e8d6 WatchSource:0}: Error finding container 73444df967788cd0a0b71d484a2ae265da3589ad4c2e3542e0fd7e7a63e0e8d6: Status 404 returned error can't find the container with id 73444df967788cd0a0b71d484a2ae265da3589ad4c2e3542e0fd7e7a63e0e8d6 Nov 01 15:47:35.760842 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.760733 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jv28d-6kkx9"] Nov 01 15:47:35.765252 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:35.765215 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8620858_3d8f_4222_abf0_0b468a12c8d9.slice/crio-7e840803a5f170f4f599be0b9523559e3111aaf8d584fe3cad6e0dda3cdc168a WatchSource:0}: Error finding container 7e840803a5f170f4f599be0b9523559e3111aaf8d584fe3cad6e0dda3cdc168a: Status 404 returned error can't find the container with id 7e840803a5f170f4f599be0b9523559e3111aaf8d584fe3cad6e0dda3cdc168a Nov 01 15:47:35.765576 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.765327 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.768993 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.767474 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvz7\" (UniqueName: \"kubernetes.io/projected/4bf6473d-8d38-41da-9d38-d71ddc4957ad-kube-api-access-9fvz7\") pod \"migrator-66df5d496b-fq2w4\" (UID: \"4bf6473d-8d38-41da-9d38-d71ddc4957ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" Nov 01 15:47:35.785516 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.785497 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvz7\" (UniqueName: \"kubernetes.io/projected/4bf6473d-8d38-41da-9d38-d71ddc4957ad-kube-api-access-9fvz7\") pod \"migrator-66df5d496b-fq2w4\" (UID: \"4bf6473d-8d38-41da-9d38-d71ddc4957ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" Nov 01 15:47:35.828277 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.828215 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2g5v9" event={"ID":"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1","Type":"ContainerStarted","Data":"73444df967788cd0a0b71d484a2ae265da3589ad4c2e3542e0fd7e7a63e0e8d6"} Nov 01 15:47:35.829202 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.829182 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" event={"ID":"a8620858-3d8f-4222-abf0-0b468a12c8d9","Type":"ContainerStarted","Data":"7e840803a5f170f4f599be0b9523559e3111aaf8d584fe3cad6e0dda3cdc168a"} Nov 01 15:47:35.868621 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.868595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.868728 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.868634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:35.868792 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.868745 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:35.868831 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.868791 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 15:47:35.868869 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.868837 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.868817402 +0000 UTC m=+35.817635324 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:35.868869 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:35.868851 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:36.868845344 +0000 UTC m=+35.817663247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : configmap "kube-rbac-proxy" not found Nov 01 15:47:35.885413 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.885389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.900298 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.900278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" Nov 01 15:47:35.945273 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.945248 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-6hrlt"] Nov 01 15:47:35.947525 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:35.947503 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff1ca6f5_1f79_49ec_954c_a8bfa9928609.slice/crio-48a8581a2509020bac9b059122692cb7b74bc22259da124fc36024a9d2074b60 WatchSource:0}: Error finding container 48a8581a2509020bac9b059122692cb7b74bc22259da124fc36024a9d2074b60: Status 404 returned error can't find the container with id 48a8581a2509020bac9b059122692cb7b74bc22259da124fc36024a9d2074b60 Nov 01 15:47:35.985457 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.985427 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2xm97"] Nov 01 15:47:35.986731 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:35.986701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:35.989474 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:35.989447 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ef09c4_10c5_4507_98c8_c94ec38f46f7.slice/crio-119e068d259449e31e788e9973d87d79b6e98ae7a40c3e89c4d8ef99d4cb1e77 WatchSource:0}: Error finding container 119e068d259449e31e788e9973d87d79b6e98ae7a40c3e89c4d8ef99d4cb1e77: Status 404 returned error can't find the container with id 119e068d259449e31e788e9973d87d79b6e98ae7a40c3e89c4d8ef99d4cb1e77 Nov 01 15:47:36.189265 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.189167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:36.199475 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.199444 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4"] Nov 01 15:47:36.202504 ip-10-0-141-145 kubenswrapper[2579]: W1101 15:47:36.202474 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf6473d_8d38_41da_9d38_d71ddc4957ad.slice/crio-cceb6f22eb157fc6cc9b3661e59cb40aca38e1549838ed0b5794f4907cf6acb3 WatchSource:0}: Error finding container cceb6f22eb157fc6cc9b3661e59cb40aca38e1549838ed0b5794f4907cf6acb3: Status 404 returned error can't find the container with id cceb6f22eb157fc6cc9b3661e59cb40aca38e1549838ed0b5794f4907cf6acb3 Nov 01 15:47:36.733167 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.733098 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:36.833252 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.833219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:36.837821 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.837783 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2xm97" event={"ID":"36ef09c4-10c5-4507-98c8-c94ec38f46f7","Type":"ContainerStarted","Data":"119e068d259449e31e788e9973d87d79b6e98ae7a40c3e89c4d8ef99d4cb1e77"} Nov 01 15:47:36.839876 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.839829 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" event={"ID":"4bf6473d-8d38-41da-9d38-d71ddc4957ad","Type":"ContainerStarted","Data":"cceb6f22eb157fc6cc9b3661e59cb40aca38e1549838ed0b5794f4907cf6acb3"} Nov 01 15:47:36.841773 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.841704 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" event={"ID":"ff1ca6f5-1f79-49ec-954c-a8bfa9928609","Type":"ContainerStarted","Data":"48a8581a2509020bac9b059122692cb7b74bc22259da124fc36024a9d2074b60"} Nov 01 15:47:36.877877 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.877841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:36.878011 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:36.877901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:36.878011 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:36.878001 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 15:47:36.878124 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:36.878063 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:38.878043016 +0000 UTC m=+37.826860933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : configmap "kube-rbac-proxy" not found Nov 01 15:47:36.878513 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:36.878491 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:36.878598 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:36.878545 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:38.878531381 +0000 UTC m=+37.827349284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:38.190072 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:38.190031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:38.190439 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:38.190248 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:38.190439 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:38.190314 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls podName:370dbf42-367c-48ac-abb8-86c01bfe2b54 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:46.190296424 +0000 UTC m=+45.139114345 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls") pod "dns-default-gx6wg" (UID: "370dbf42-367c-48ac-abb8-86c01bfe2b54") : secret "dns-default-metrics-tls" not found Nov 01 15:47:38.331357 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:38.331315 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:38.849869 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:38.849832 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" event={"ID":"4bf6473d-8d38-41da-9d38-d71ddc4957ad","Type":"ContainerStarted","Data":"e5f134618ce8405a4a7ff49afacea0e7b4e51e169f6ae746b54381e48c118110"} Nov 01 15:47:38.865860 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:38.865788 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" podStartSLOduration=1.765403081 podStartE2EDuration="3.865768873s" totalImagesPullingTime="2.100365792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:38.865110138 +0000 UTC m=+37.813928081" watchObservedRunningTime="2026-11-01 15:47:38.865768873 +0000 UTC m=+37.814586794" Nov 01 15:47:38.895734 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:38.895708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:38.895895 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:38.895769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:38.896177 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:38.896157 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:38.896237 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:38.896219 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.896203139 +0000 UTC m=+41.845021056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:38.896599 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:38.896581 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 01 15:47:38.896705 ip-10-0-141-145 kubenswrapper[2579]: E1101 15:47:38.896630 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:42.896618574 +0000 UTC m=+41.845436497 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : configmap "kube-rbac-proxy" not found Nov 01 15:47:39.854774 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:39.854718 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" event={"ID":"a8620858-3d8f-4222-abf0-0b468a12c8d9","Type":"ContainerStarted","Data":"8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742"} Nov 01 15:47:39.856839 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:39.856804 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" event={"ID":"ff1ca6f5-1f79-49ec-954c-a8bfa9928609","Type":"ContainerStarted","Data":"cefc559360ac10fc50c53283661aedcc4dff039e48e895705aa2c52de42846b4"} Nov 01 15:47:39.870867 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:39.870815 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" podStartSLOduration=1.640348821 podStartE2EDuration="4.870799005s" totalImagesPullingTime="3.230450184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:39.869989121 +0000 UTC m=+38.818807051" watchObservedRunningTime="2026-11-01 15:47:39.870799005 +0000 UTC m=+38.819616933" Nov 01 15:47:39.887071 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:39.887020 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" podStartSLOduration=1.144094004 podStartE2EDuration="4.887003881s" totalImagesPullingTime="3.742909877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:39.88622222 +0000 UTC m=+38.835040148" watchObservedRunningTime="2026-11-01 15:47:39.887003881 +0000 UTC m=+38.835821810" Nov 01 15:47:40.559045 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:40.558966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:40.682096 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:40.682068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:40.785469 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:40.785433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:40.867813 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:40.861995 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2xm97" event={"ID":"36ef09c4-10c5-4507-98c8-c94ec38f46f7","Type":"ContainerStarted","Data":"92e7f3c814af6becc094a360a843def62b7f1381a3b9383c373f87a62f4ddc35"} Nov 01 15:47:40.876959 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:40.876897 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2xm97" podStartSLOduration=35.264543593 podStartE2EDuration="39.876880186s" totalImagesPullingTime="4.612336593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:47:40.876480232 +0000 UTC m=+39.825298158" watchObservedRunningTime="2026-11-01 15:47:40.876880186 +0000 UTC m=+39.825698114" Nov 01 15:47:40.884839 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:40.884809 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:41.065871 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:41.065805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:41.229662 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:41.229587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:41.828173 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:41.828144 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:41.864239 ip-10-0-141-145 kubenswrapper[2579]: I1101 15:47:41.864208 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:42.237232 ip-10-0-141-145 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 15:47:42.251156 ip-10-0-141-145 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 15:47:42.251320 ip-10-0-141-145 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 15:47:42.251382 ip-10-0-141-145 systemd[1]: kubelet.service: Consumed 2.433s CPU time. Nov 01 15:47:42.268204 ip-10-0-141-145 systemd[1]: Starting Kubernetes Kubelet... Nov 01 15:47:42.372292 ip-10-0-141-145 kubenswrapper[4573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:42.372292 ip-10-0-141-145 kubenswrapper[4573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 15:47:42.372292 ip-10-0-141-145 kubenswrapper[4573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:42.372292 ip-10-0-141-145 kubenswrapper[4573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 15:47:42.374346 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374266 4573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:42.374346 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374345 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374350 4573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374354 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374357 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374360 4573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374363 4573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374365 4573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374368 4573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374371 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374374 4573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374376 4573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374379 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374382 4573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374384 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374387 4573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374397 4573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374400 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374403 4573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374405 4573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:42.374417 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374408 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374411 4573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374414 4573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374416 4573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374420 4573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374423 4573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374426 4573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374429 4573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374432 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374435 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374437 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374440 4573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374443 4573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374446 4573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374448 4573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374452 4573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374456 4573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374460 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374463 4573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:42.374889 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374465 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374468 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374471 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374473 4573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374476 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374478 4573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374481 4573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374483 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374486 4573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374489 4573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374492 4573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374496 4573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374498 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374501 4573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374504 4573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374506 4573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374509 4573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374512 4573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374514 4573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374517 4573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:42.375504 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374521 4573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374523 4573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374526 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374528 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374531 4573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374534 4573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374536 4573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374539 4573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374541 4573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374544 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374546 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374549 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374551 4573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374554 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374557 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374559 4573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374563 4573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374565 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374568 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374570 4573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:42.376171 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374573 4573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374576 4573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374579 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374582 4573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374585 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374587 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374591 4573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374595 4573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374598 4573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374601 4573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374603 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374606 4573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374608 4573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374611 4573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374614 4573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374616 4573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374619 4573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374622 4573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374625 4573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374628 4573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:42.376682 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374630 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374633 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374635 4573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374638 4573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374640 4573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374643 4573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.374646 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375409 4573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375416 4573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375418 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375422 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375425 4573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375428 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375431 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375434 4573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375438 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375441 4573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375444 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375447 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:42.377185 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375450 4573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375453 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375456 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375458 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375461 4573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375464 4573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375466 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375469 4573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375472 4573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375474 4573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375477 4573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375479 4573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375482 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375486 4573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375488 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375491 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375493 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375496 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375498 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375501 4573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:42.377640 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375506 4573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375509 4573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375512 4573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375514 4573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375517 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375520 4573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375522 4573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375525 4573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375529 4573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375531 4573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375534 4573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375537 4573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375540 4573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375542 4573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375545 4573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375548 4573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375550 4573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375553 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375556 4573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375559 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:42.378137 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375561 4573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375564 4573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375566 4573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375569 4573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375572 4573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375574 4573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375577 4573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375582 4573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375585 4573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375588 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375590 4573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375593 4573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375597 4573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375600 4573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375602 4573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375605 4573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375607 4573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375610 4573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375613 4573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375615 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:42.378621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375618 4573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375621 4573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375626 4573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375629 4573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375632 4573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375635 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375638 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375640 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375643 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375646 4573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375649 4573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375651 4573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375654 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375656 4573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375659 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375662 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375664 4573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375667 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375669 4573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:42.379166 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375673 4573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375676 4573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375678 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375681 4573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375683 4573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375687 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375690 4573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375693 4573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375696 4573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375699 4573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375703 4573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375708 4573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375712 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375715 4573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.375717 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376216 4573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376223 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376226 4573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376229 4573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:42.379621 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376232 4573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376235 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376238 4573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376240 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376243 4573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376246 4573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376248 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376251 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376254 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376257 4573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376260 4573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376262 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376265 4573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376270 4573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376272 4573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376275 4573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376278 4573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376280 4573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376284 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376287 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:42.380099 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376290 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376292 4573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376295 4573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376297 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376300 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376303 4573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376305 4573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376308 4573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376311 4573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376314 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376317 4573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376319 4573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376322 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376324 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376327 4573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376330 4573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376332 4573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376335 4573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376337 4573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376340 4573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:42.380613 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376343 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376345 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376348 4573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376351 4573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376353 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376357 4573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376360 4573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376362 4573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376365 4573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376367 4573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376371 4573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376374 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376377 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376380 4573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376382 4573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376385 4573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376388 4573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376390 4573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376393 4573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:42.381101 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376398 4573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376402 4573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376406 4573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376409 4573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376411 4573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376414 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376417 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376419 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376422 4573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376424 4573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376427 4573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376429 4573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376432 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376435 4573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376437 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376440 4573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376443 4573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376445 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376452 4573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376455 4573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:42.381687 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376458 4573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376461 4573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376464 4573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376468 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376471 4573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376473 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376476 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376478 4573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376481 4573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376484 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376486 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376489 4573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376492 4573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376494 4573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376497 4573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376501 4573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376503 4573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376506 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376509 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:42.382224 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376512 4573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376515 4573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376517 4573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.376520 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376613 4573 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376628 4573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376637 4573 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376642 4573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376646 4573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376650 4573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376654 4573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376660 4573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376663 4573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376666 4573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376670 4573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376673 4573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376678 4573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376681 4573 flags.go:64] FLAG: --cgroup-root="" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376684 4573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376687 4573 flags.go:64] FLAG: --client-ca-file="" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376690 4573 flags.go:64] FLAG: --cloud-provider="external" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376693 4573 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376697 4573 flags.go:64] FLAG: --cluster-domain="" Nov 01 15:47:42.382689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376700 4573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376703 4573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376707 4573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376710 4573 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376714 4573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376717 4573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376721 4573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376724 4573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376727 4573 flags.go:64] FLAG: --contention-profiling="false" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376731 4573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376734 4573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376738 4573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376741 4573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376745 4573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376763 4573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376768 4573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376773 4573 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376778 4573 flags.go:64] FLAG: --enable-server="true" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376782 4573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376789 4573 flags.go:64] FLAG: --event-burst="100" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376794 4573 flags.go:64] FLAG: --event-qps="50" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376797 4573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376801 4573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376804 4573 flags.go:64] FLAG: --eviction-hard="" Nov 01 15:47:42.383296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376808 4573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376813 4573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376816 4573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376819 4573 flags.go:64] FLAG: --eviction-soft="" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376823 4573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376826 4573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376829 4573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376832 4573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376835 4573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376838 4573 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376841 4573 flags.go:64] FLAG: --feature-gates="" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376845 4573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376849 4573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376852 4573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376855 4573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376859 4573 flags.go:64] FLAG: --healthz-port="10248" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376862 4573 flags.go:64] FLAG: --help="false" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376865 4573 flags.go:64] FLAG: --hostname-override="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376868 4573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376872 4573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376875 4573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376878 4573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376881 4573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 15:47:42.383930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376884 4573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376887 4573 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376890 4573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376893 4573 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376896 4573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376900 4573 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376903 4573 flags.go:64] FLAG: --kube-reserved="" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376906 4573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376909 4573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376912 4573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376916 4573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376919 4573 flags.go:64] FLAG: --lock-file="" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376922 4573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376925 4573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376928 4573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376933 4573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376937 4573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376940 4573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376943 4573 flags.go:64] FLAG: --logging-format="text" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376946 4573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376950 4573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376953 4573 flags.go:64] FLAG: --manifest-url="" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376956 4573 flags.go:64] FLAG: --manifest-url-header="" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376960 4573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376963 4573 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 15:47:42.384629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376968 4573 flags.go:64] FLAG: --max-pods="110" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376971 4573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376974 4573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376977 4573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376980 4573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376983 4573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376986 4573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376989 4573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376996 4573 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.376999 4573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377002 4573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377005 4573 flags.go:64] FLAG: --pod-cidr="" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377008 4573 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377012 4573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377015 4573 flags.go:64] FLAG: --pods-per-core="0" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377018 4573 flags.go:64] FLAG: --port="10250" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377021 4573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377025 4573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-030866921209e4fe1" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377028 4573 flags.go:64] FLAG: --qos-reserved="" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377032 4573 flags.go:64] FLAG: --read-only-port="10255" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377035 4573 flags.go:64] FLAG: --register-node="true" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377038 4573 flags.go:64] FLAG: --register-with-taints="" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377042 4573 flags.go:64] FLAG: --registry-burst="10" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377045 4573 flags.go:64] FLAG: --registry-qps="5" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377048 4573 flags.go:64] FLAG: --reserved-cpus="" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377051 4573 flags.go:64] FLAG: --reserved-memory="" Nov 01 15:47:42.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377055 4573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377058 4573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377061 4573 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377064 4573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377068 4573 flags.go:64] FLAG: --runonce="false" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377071 4573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377074 4573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377077 4573 flags.go:64] FLAG: --seccomp-default="false" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377081 4573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377084 4573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377087 4573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377090 4573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377093 4573 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377096 4573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377099 4573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377102 4573 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377105 4573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377108 4573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377111 4573 flags.go:64] FLAG: --system-cgroups="" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377114 4573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377120 4573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377122 4573 flags.go:64] FLAG: --tls-cert-file="" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377125 4573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377130 4573 flags.go:64] FLAG: --tls-min-version="" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377133 4573 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 15:47:42.386366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377136 4573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377139 4573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377142 4573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377145 4573 flags.go:64] FLAG: --v="2" Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377149 4573 flags.go:64] FLAG: --version="false" Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377153 4573 flags.go:64] FLAG: --vmodule="" Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377157 4573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377160 4573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377321 4573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377327 4573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377330 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377332 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377335 4573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377338 4573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377341 4573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377350 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377353 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377357 4573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377361 4573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377363 4573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377366 4573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:42.387251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377369 4573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377372 4573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377374 4573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377377 4573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377380 4573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377383 4573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377385 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377388 4573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377391 4573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377394 4573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377397 4573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377399 4573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377402 4573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377405 4573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377407 4573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377410 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377412 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377415 4573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377418 4573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377420 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:42.387778 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377423 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377427 4573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377431 4573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377434 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377437 4573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377439 4573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377442 4573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377450 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377455 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377457 4573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377460 4573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377463 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377465 4573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377468 4573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377471 4573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377473 4573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377476 4573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377484 4573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377487 4573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:42.388519 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377490 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377492 4573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377495 4573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377498 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377501 4573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377503 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377506 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377509 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377511 4573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377514 4573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377517 4573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377520 4573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377524 4573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377527 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377531 4573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377533 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377536 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377539 4573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377541 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:42.389068 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377544 4573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377546 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377555 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377558 4573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377561 4573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377564 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377566 4573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377569 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377572 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377574 4573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377577 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377580 4573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377582 4573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377585 4573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377588 4573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377591 4573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377593 4573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377596 4573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377599 4573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377601 4573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:42.389599 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377604 4573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377606 4573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377609 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377612 4573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377615 4573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377617 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377620 4573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377622 4573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377625 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377628 4573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377630 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377633 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377635 4573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377638 4573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377643 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.377676 4573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:42.390103 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377719 4573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377723 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377726 4573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377729 4573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377732 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377735 4573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377738 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377741 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377744 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377746 4573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377763 4573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377768 4573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377773 4573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377778 4573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377782 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377785 4573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377787 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377790 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377793 4573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:42.390646 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377795 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377798 4573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377801 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377803 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377806 4573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377808 4573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377811 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377814 4573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377816 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377819 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377821 4573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377825 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377867 4573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377872 4573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377875 4573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377878 4573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377881 4573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377886 4573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377888 4573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377891 4573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:42.391218 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377894 4573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377897 4573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377899 4573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377902 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377905 4573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377909 4573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377918 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377922 4573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377925 4573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377927 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377930 4573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377933 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377935 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377938 4573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377941 4573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377943 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377946 4573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377949 4573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377951 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:42.391961 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377954 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377957 4573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377959 4573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377962 4573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377964 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377968 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377971 4573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377974 4573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377977 4573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377980 4573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377984 4573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377987 4573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377990 4573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377992 4573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377995 4573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.377998 4573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378000 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378003 4573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378006 4573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:42.392471 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378008 4573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378011 4573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378013 4573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378016 4573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378019 4573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378021 4573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378024 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378027 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378029 4573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378032 4573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378035 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378037 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378042 4573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378046 4573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378050 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378053 4573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378055 4573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378058 4573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378062 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:42.393012 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378065 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378067 4573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378070 4573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378073 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378078 4573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378081 4573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378084 4573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378086 4573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378089 4573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.378092 4573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.381711 4573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.381725 4573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.381744 4573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.381768 4573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381824 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381829 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381833 4573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381836 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381839 4573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381842 4573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:42.393533 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381845 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381860 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381865 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381868 4573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381872 4573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381876 4573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381879 4573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381883 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381886 4573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381889 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381892 4573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381896 4573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381899 4573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381901 4573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381905 4573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381908 4573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381911 4573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381922 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381925 4573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:42.394131 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381928 4573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381931 4573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381934 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381936 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381939 4573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381942 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381944 4573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381947 4573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381950 4573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381952 4573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381955 4573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381958 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381961 4573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381964 4573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381967 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381970 4573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381972 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381975 4573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381977 4573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:42.394660 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381980 4573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381983 4573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381986 4573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381989 4573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381991 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381994 4573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381997 4573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.381999 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382002 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382005 4573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382008 4573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382011 4573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382015 4573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382019 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382022 4573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382025 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382027 4573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382030 4573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382032 4573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:42.395197 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382035 4573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382038 4573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382040 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382043 4573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382045 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382048 4573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382051 4573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382053 4573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382056 4573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382059 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382062 4573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382065 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382068 4573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382070 4573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382073 4573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382075 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382078 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382081 4573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382083 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382086 4573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:42.395703 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382088 4573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382091 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382094 4573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382097 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382100 4573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382102 4573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382105 4573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382108 4573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382111 4573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382113 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382116 4573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382119 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382121 4573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382124 4573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382126 4573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382129 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382131 4573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382134 4573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382137 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:42.396245 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382140 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382142 4573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382145 4573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382148 4573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.382175 4573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382305 4573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382310 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382313 4573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382316 4573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382319 4573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382322 4573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382324 4573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382327 4573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382330 4573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382332 4573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:42.397013 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382335 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382338 4573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382341 4573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382344 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382347 4573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382350 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382352 4573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382355 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382357 4573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382360 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382363 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382365 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382368 4573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382370 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382373 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382376 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382378 4573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382381 4573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382384 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382386 4573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:42.397607 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382389 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382392 4573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382395 4573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382397 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382400 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382402 4573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382405 4573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382407 4573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382410 4573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382413 4573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382415 4573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382418 4573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382420 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382422 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382425 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382428 4573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382431 4573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382434 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382436 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382439 4573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:42.398463 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382442 4573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382444 4573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382447 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382450 4573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382453 4573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382455 4573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382458 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382460 4573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382463 4573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382466 4573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382468 4573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382471 4573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382474 4573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382476 4573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382479 4573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382481 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382484 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382487 4573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382489 4573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382492 4573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:42.399367 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382494 4573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382498 4573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382502 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382506 4573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382508 4573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382511 4573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382515 4573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382518 4573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382522 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382525 4573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382528 4573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382531 4573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382534 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382536 4573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382539 4573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382541 4573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382544 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382547 4573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:42.400179 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382549 4573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382552 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382554 4573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382559 4573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382562 4573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382564 4573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382567 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382570 4573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382573 4573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382575 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382578 4573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382581 4573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382583 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382586 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382589 4573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382591 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382594 4573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:42.400938 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382596 4573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.382619 4573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382649 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382653 4573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382656 4573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382659 4573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382662 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382666 4573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382668 4573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382671 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382673 4573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382676 4573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382679 4573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382681 4573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382684 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 15:47:42.401611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382687 4573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382690 4573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382693 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382696 4573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382700 4573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382702 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382705 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382708 4573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382711 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382714 4573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382716 4573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382719 4573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382722 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382725 4573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382728 4573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382730 4573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382733 4573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382736 4573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382738 4573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 15:47:42.402242 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382741 4573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382743 4573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382758 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382763 4573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382767 4573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382770 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382773 4573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382775 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382778 4573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382781 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382784 4573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382786 4573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382789 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382791 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382794 4573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382797 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382799 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382804 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382807 4573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382809 4573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 15:47:42.403009 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382812 4573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382815 4573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382818 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382821 4573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382823 4573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382826 4573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382829 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382832 4573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382834 4573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382837 4573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382840 4573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382842 4573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382845 4573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382847 4573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382850 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382853 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382855 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382858 4573 feature_gate.go:386] unrecognized feature gate: Example Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382861 4573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382864 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 15:47:42.403827 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382867 4573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382869 4573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382872 4573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382874 4573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382877 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382880 4573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382883 4573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382885 4573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382888 4573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382892 4573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382895 4573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382898 4573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382900 4573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382903 4573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382906 4573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382909 4573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382912 4573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382914 4573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382917 4573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382919 4573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 15:47:42.404628 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382922 4573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382925 4573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382927 4573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382931 4573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382935 4573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382938 4573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382941 4573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382943 4573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382948 4573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382952 4573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382955 4573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382959 4573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382963 4573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:42.382967 4573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.383016 4573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.383989 4573 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.384038 4573 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.384208 4573 server.go:1014] "Starting client certificate rotation" Nov 01 15:47:42.405450 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.384365 4573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 15:47:42.406189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.384413 4573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 15:42:01 +0000 UTC" deadline="2028-07-20 05:49:43.650609872 +0000 UTC" Nov 01 15:47:42.406189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.384449 4573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15038h2m1.266163413s" Nov 01 15:47:42.406189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.384792 4573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:42.406189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.387292 4573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 15:47:42.406189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.388890 4573 log.go:25] "Validated CRI v1 runtime API" Nov 01 15:47:42.406189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.390469 4573 log.go:25] "Validated CRI v1 image API" Nov 01 15:47:42.406189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.391231 4573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 15:47:42.406189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.393269 4573 fs.go:135] Filesystem UUIDs: map[0c52f548-7fd8-4640-965b-ca34386c28a7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 fde469a6-9826-4d71-8302-4d99deeb5887:/dev/nvme0n1p3] Nov 01 15:47:42.406547 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.393288 4573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/10f5ee1b941f2c53e73e955e1092ccfca4a2a2ff935b6c2763c7e1100b244af0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/10f5ee1b941f2c53e73e955e1092ccfca4a2a2ff935b6c2763c7e1100b244af0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/119e068d259449e31e788e9973d87d79b6e98ae7a40c3e89c4d8ef99d4cb1e77/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/119e068d259449e31e788e9973d87d79b6e98ae7a40c3e89c4d8ef99d4cb1e77/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48a8581a2509020bac9b059122692cb7b74bc22259da124fc36024a9d2074b60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48a8581a2509020bac9b059122692cb7b74bc22259da124fc36024a9d2074b60/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4a32f04ab0b6239f10d269bfc5393003e4673900e8be00c3616073d1fce2c96d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4a32f04ab0b6239f10d269bfc5393003e4673900e8be00c3616073d1fce2c96d/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5a0675364f6c2fe46751d9b52f2d55c85ecacb060b93fac6a8fea44471d46edc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5a0675364f6c2fe46751d9b52f2d55c85ecacb060b93fac6a8fea44471d46edc/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6121d84167620035b567b29c921b42e0a1fe83d4a7ca7fa8abd028a817fc4275/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6121d84167620035b567b29c921b42e0a1fe83d4a7ca7fa8abd028a817fc4275/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c44377411aba96da3f9ee0bdd884bcd7bed5bfbed5bb5b593c533321525218b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c44377411aba96da3f9ee0bdd884bcd7bed5bfbed5bb5b593c533321525218b/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73444df967788cd0a0b71d484a2ae265da3589ad4c2e3542e0fd7e7a63e0e8d6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73444df967788cd0a0b71d484a2ae265da3589ad4c2e3542e0fd7e7a63e0e8d6/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e840803a5f170f4f599be0b9523559e3111aaf8d584fe3cad6e0dda3cdc168a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e840803a5f170f4f599be0b9523559e3111aaf8d584fe3cad6e0dda3cdc168a/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9124c1b00ef6d2bc43afdb2d4033b9d6dcaf936bb18c4c930ca4dc5bcb2ffaad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9124c1b00ef6d2bc43afdb2d4033b9d6dcaf936bb18c4c930ca4dc5bcb2ffaad/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9388c520390b9b575fccf68a2923ba58f60cce989562911521a12f5131f97c4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9388c520390b9b575fccf68a2923ba58f60cce989562911521a12f5131f97c4a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ba4251a2aab9ac5f477e533e1061c75fe685c92bc68877976888934f220d0679/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ba4251a2aab9ac5f477e533e1061c75fe685c92bc68877976888934f220d0679/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7273e3bd74ed6a5a2ba8093cba50f15427e53bf06692650840370edefaff1ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7273e3bd74ed6a5a2ba8093cba50f15427e53bf06692650840370edefaff1ce/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cceb6f22eb157fc6cc9b3661e59cb40aca38e1549838ed0b5794f4907cf6acb3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cceb6f22eb157fc6cc9b3661e59cb40aca38e1549838ed0b5794f4907cf6acb3/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d97ad3dd117cbd1d90647fcd150f0dd4cb81c1c8d851d9dc54852f7fd840b0d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d97ad3dd117cbd1d90647fcd150f0dd4cb81c1c8d851d9dc54852f7fd840b0d5/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9485f27d362087ade8dd7c28848a97cdbeae94df836cdb8870ba259cf176a39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9485f27d362087ade8dd7c28848a97cdbeae94df836cdb8870ba259cf176a39/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~projected/kube-api-access-blnvk:{mountpoint:/var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~projected/kube-api-access-blnvk major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06506c1c-88b0-42e1-ab42-8cc842decd18/volumes/kubernetes.io~projected/kube-api-access-rgr87:{mountpoint:/var/lib/kubelet/pods/06506c1c-88b0-42e1-ab42-8cc842decd18/volumes/kubernetes.io~projected/kube-api-access-rgr87 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a2d0642-db9c-4bd7-a804-51167fdd62fd/volumes/kubernetes.io~projected/kube-api-access-4s446:{mountpoint:/var/lib/kubelet/pods/0a2d0642-db9c-4bd7-a804-51167fdd62fd/volumes/kubernetes.io~projected/kube-api-access-4s446 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36ef09c4-10c5-4507-98c8-c94ec38f46f7/volumes/kubernetes.io~projected/kube-api-access-nzdnx:{mountpoint:/var/lib/kubelet/pods/36ef09c4-10c5-4507-98c8-c94ec38f46f7/volumes/kubernetes.io~projected/kube-api-access-nzdnx major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/370dbf42-367c-48ac-abb8-86c01bfe2b54/volumes/kubernetes.io~projected/kube-api-access-9nrjm:{mountpoint:/var/lib/kubelet/pods/370dbf42-367c-48ac-abb8-86c01bfe2b54/volumes/kubernetes.io~projected/kube-api-access-9nrjm major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4926376c-9c72-4d94-8505-959c770493a8/volumes/kubernetes.io~projected/kube-api-access-cksfz:{mountpoint:/var/lib/kubelet/pods/4926376c-9c72-4d94-8505-959c770493a8/volumes/kubernetes.io~projected/kube-api-access-cksfz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volumes/kubernetes.io~projected/kube-api-access-vwkgh:{mountpoint:/var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volumes/kubernetes.io~projected/kube-api-access-vwkgh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4bf6473d-8d38-41da-9d38-d71ddc4957ad/volumes/kubernetes.io~projected/kube-api-access-9fvz7:{mountpoint:/var/lib/kubelet/pods/4bf6473d-8d38-41da-9d38-d71ddc4957ad/volumes/kubernetes.io~projected/kube-api-access-9fvz7 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9e4381-b8a2-4d0d-ade3-75df4d07810f/volumes/kubernetes.io~projected/kube-api-access-brr4h:{mountpoint:/var/lib/kubelet/pods/8b9e4381-b8a2-4d0d-ade3-75df4d07810f/volumes/kubernetes.io~projected/kube-api-access-brr4h major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b9e4381-b8a2-4d0d-ade3-75df4d07810f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8b9e4381-b8a2-4d0d-ade3-75df4d07810f/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a22ea5ba-32e9-4552-802d-d11594d04f29/volumes/kubernetes.io~projected/kube-api-access-7b7q6:{mountpoint:/var/lib/kubelet/pods/a22ea5ba-32e9-4552-802d-d11594d04f29/volumes/kubernetes.io~projected/kube-api-access-7b7q6 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8620858-3d8f-4222-abf0-0b468a12c8d9/volumes/kubernetes.io~projected/kube-api-access-pfd6r:{mountpoint:/var/lib/kubelet/pods/a8620858-3d8f-4222-abf0-0b468a12c8d9/volumes/kubernetes.io~projected/kube-api-access-pfd6r major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ada3c550-b52f-4236-88db-a07ed2c6fdda/volumes/kubernetes.io~projected/kube-api-access-6rzcv:{mountpoint:/var/lib/kubelet/pods/ada3c550-b52f-4236-88db-a07ed2c6fdda/volumes/kubernetes.io~projected/kube-api-access-6rzcv major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da994e56-8f1d-4f00-893a-670984816adc/volumes/kubernetes.io~projected/kube-api-access-rn6rl:{mountpoint:/var/lib/kubelet/pods/da994e56-8f1d-4f00-893a-670984816adc/volumes/kubernetes.io~projected/kube-api-access-rn6rl major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8a2984b-e706-43f8-bd87-dd5bc809c36e/volumes/kubernetes.io~projected/kube-api-access-dd7gp:{mountpoint:/var/lib/kubelet/pods/e8a2984b-e706-43f8-bd87-dd5bc809c36e/volumes/kubernetes.io~projected/kube-api-access-dd7gp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/edc57054-ce41-4e4e-97f6-b127bf151d23/volumes/kubernetes.io~projected/kube-api-access-djclq:{mountpoint:/var/lib/kubelet/pods/edc57054-ce41-4e4e-97f6-b127bf151d23/volumes/kubernetes.io~projected/kube-api-access-djclq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f7e5339a-b073-4184-9acc-5a701276ea1c/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/f7e5339a-b073-4184-9acc-5a701276ea1c/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff1ca6f5-1f79-49ec-954c-a8bfa9928609/volumes/kubernetes.io~projected/kube-api-access-d4sb5:{mountpoint:/var/lib/kubelet/pods/ff1ca6f5-1f79-49ec-954c-a8bfa9928609/volumes/kubernetes.io~projected/kube-api-access-d4sb5 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ff1ca6f5-1f79-49ec-954c-a8bfa9928609/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/ff1ca6f5-1f79-49ec-954c-a8bfa9928609/volumes/kubernetes.io~secret/signing-key major:0 minor:220 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/b5465c5155b6f61a4cba1b376f684d1ce707f5996eb499ac35d2b02220bb1a7a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e7a21f96c5ab7cf02b8ffb8779bf64805ffe7afd1d2e6ab0c4065892a6439dbc/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0754801d6023ef5dd1cad56ea33af21d6565da60362b7c280eef10e03222c2d3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/06a025caf10eff92991cc7b7615a29a42384d7374dee38370cda0daeff970049/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2e2ae277feddfa97941ea5912436e250b4e7a44111f96b6c43f3e4d46bbd4f86/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b213d1a65742b2ee954b55aa37055271d5f2c0ce56375350f90edb54d54bfd07/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9a537e2ef8919092f60b3dcc8d6f0915dd4ceb7749b260ae393288e535c5c788/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c9f1cd8850e6c4d53a3badb8edaa794bfdf6fc972c00146f229d9a02c6aadf14/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e10194751d5bf36cb89d3bdc7483d1cfbd0168bbf0472f45e8f1d8ce42678274/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/14da7f621859b3feabf1cc391382546f9a64f731e35b75fb952a8d020722dd58/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/ba73462fa902062a77e742f7ce81ccede7caf3e17db887d0e14a8a0b745b9d75/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/308a02561a3809206cb504e781999abf97e08713495b78efb6d6a1ba934a6cbe/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/b68a016357b88736b32d66cb46c026dca4447945e5eac0755360194533247286/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/544c3cc446c12ac11e150fb4ba9e0f9f80a36e8b1286f973138c173ee1e5a02c/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9f590d3c6d5496d9c4e003acbdf8fc26f46ad5e4fbb66e010ba0b8446d797d84/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/ecd59276f54633324cf4a0ae3ca9ba6ef04ef2c7fb6c393f12fc4931292343dc/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2cf12a3abbac30ca1911128463621fe89d3d20eade1ad4fc2df9d94341bc14c0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5d0c23dedfc2f312ababe17660f685aaab5122b3849b9287c2b5341bf3f7b367/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/646d22499c6adff5b0ffbbd359b7fb147eda028753be7e432be510744aa4d2d8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/7b219da35a0fe96c5a72a187a96bbe5fd87d77b21164db2f55afe8179c373792/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8b8732b2195ca5470396c55c083bbaeaacdd362f90ad1550a8d0ade5e99dad99/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/10cb730ae9342e05bc29e4e2383bead15ecebe1a161bdb3d575a104039a13be5/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/18a25dc80cd81879e9bae6598b642dd9d2b821ad0a39492a88cefb946069427b/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/0d7526278773d935a60fb0ffebbe4039ce1c927673144fcd4451f82263cd13e6/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/19f26bf8963b0c69022b3989864759efab464ace65ae8b940747ccfef2f2f4d7/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/0289dcb357059bba8a7db11db9545d4863af2ef3e24c6c84746ae17e50bb350a/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/da9eb61afed82bc2f025cd59cc5e1f4cb6e659d6bd50c5937974b10b9ce8ff23/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/aca500ef6ea767853cf3bb3ad7d431ff65623b9d6a61508d99151bad2e853909/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-239:{mountpoint:/var/lib/containers/storage/overlay/c3a562662e8311071634a8ad5d9d9d8df6070f5fc14db3a7accc5fc3161d7b1a/merged major:0 minor:239 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/516383f3fa7a1cf5c2a4eea29e952b9b70e233e41d8d3a81893726ac05286436/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/d0e118c0528de12764d58fbdeff2e6bef9143893ed5f6d8e156cb2cbc78aea06/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/3917a8bd7b7be9da8f4928bbd6f9b5f503fbebde6016fdc8310b10fb9f4c9aba/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/69acafa31574e71f1e9df4a215acbb30d54122699727485b9a3293cf6753bcd6/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/3b5f06dd93aea6f2684ad59a903c91c0fd0a6b7ad5ccc13ca0681cb8526ed303/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/fb6e62422f05969a051a1f11723182e6da165abc96f43d61b0e04409b1d6ae4d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/70e7d187db95ca3203a8c3ab96e2973a31f7a70c9b16422d1fbb811ba96c095b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/9b84b23d2532a60dd8938230314fe6b8499cf2f570265e45dbeb219d96fc1676/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/3c23de0f654d04a00c92a1dbb2db681a1265b810d178e46edcae432155d187d1/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/47b65c9e32ac4bd261d2cde91c9fd72a503a71f907ed884d57750c8b205263cd/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/505940b0f130c5dfe8dadc294429401bca8cc713c9a37273b8bfb5c10aebb752/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/278decc99e8eb9a13565a0e049cf615b20e4bb829fc6c8a0113665dfc61d0bee/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/806963724bd3f6f75d5128c48d5a992b0bc3efb6b0f61c28162d14511698de7f/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/987e33f61a7bb421f6b7ec79655d4f37a674ddfa943b8422cefe7b12e7e563f4/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.400249 4573 manager.go:217] Machine: {Timestamp:2026-11-01 15:47:42.399807526 +0000 UTC m=+0.102973455 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107571 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28e78f05160518db3a72d2f7df20f2 SystemUUID:ec28e78f-0516-0518-db3a-72d2f7df20f2 BootID:f8cba4af-e90c-4cf5-885d-4faf94a6d338 Filesystems:[{Device:/run/containers/storage/overlay-containers/4a32f04ab0b6239f10d269bfc5393003e4673900e8be00c3616073d1fce2c96d/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a8620858-3d8f-4222-abf0-0b468a12c8d9/volumes/kubernetes.io~projected/kube-api-access-pfd6r DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06506c1c-88b0-42e1-ab42-8cc842decd18/volumes/kubernetes.io~projected/kube-api-access-rgr87 DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/10f5ee1b941f2c53e73e955e1092ccfca4a2a2ff935b6c2763c7e1100b244af0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73444df967788cd0a0b71d484a2ae265da3589ad4c2e3542e0fd7e7a63e0e8d6/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48a8581a2509020bac9b059122692cb7b74bc22259da124fc36024a9d2074b60/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ba4251a2aab9ac5f477e533e1061c75fe685c92bc68877976888934f220d0679/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volumes/kubernetes.io~projected/kube-api-access-vwkgh DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9e4381-b8a2-4d0d-ade3-75df4d07810f/volumes/kubernetes.io~projected/kube-api-access-brr4h DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9485f27d362087ade8dd7c28848a97cdbeae94df836cdb8870ba259cf176a39/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da994e56-8f1d-4f00-893a-670984816adc/volumes/kubernetes.io~projected/kube-api-access-rn6rl DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e840803a5f170f4f599be0b9523559e3111aaf8d584fe3cad6e0dda3cdc168a/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ff1ca6f5-1f79-49ec-954c-a8bfa9928609/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:220 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4bf6473d-8d38-41da-9d38-d71ddc4957ad/volumes/kubernetes.io~projected/kube-api-access-9fvz7 DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7273e3bd74ed6a5a2ba8093cba50f15427e53bf06692650840370edefaff1ce/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f7e5339a-b073-4184-9acc-5a701276ea1c/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a22ea5ba-32e9-4552-802d-d11594d04f29/volumes/kubernetes.io~projected/kube-api-access-7b7q6 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9124c1b00ef6d2bc43afdb2d4033b9d6dcaf936bb18c4c930ca4dc5bcb2ffaad/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5a0675364f6c2fe46751d9b52f2d55c85ecacb060b93fac6a8fea44471d46edc/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-239 DeviceMajor:0 DeviceMinor:239 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d97ad3dd117cbd1d90647fcd150f0dd4cb81c1c8d851d9dc54852f7fd840b0d5/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ada3c550-b52f-4236-88db-a07ed2c6fdda/volumes/kubernetes.io~projected/kube-api-access-6rzcv DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ff1ca6f5-1f79-49ec-954c-a8bfa9928609/volumes/kubernetes.io~projected/kube-api-access-d4sb5 DeviceMajor:0 DeviceMinor:221 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/0a2d0642-db9c-4bd7-a804-51167fdd62fd/volumes/kubernetes.io~projected/kube-api-access-4s446 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9388c520390b9b575fccf68a2923ba58f60cce989562911521a12f5131f97c4a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c44377411aba96da3f9ee0bdd884bcd7bed5bfbed5bb5b593c533321525218b/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/119e068d259449e31e788e9973d87d79b6e98ae7a40c3e89c4d8ef99d4cb1e77/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~projected/kube-api-access-blnvk DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36ef09c4-10c5-4507-98c8-c94ec38f46f7/volumes/kubernetes.io~projected/kube-api-access-nzdnx DeviceMajor:0 DeviceMinor:217 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cceb6f22eb157fc6cc9b3661e59cb40aca38e1549838ed0b5794f4907cf6acb3/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a228ff8-0200-4b6f-8948-9e351926a3b7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4926376c-9c72-4d94-8505-959c770493a8/volumes/kubernetes.io~projected/kube-api-access-cksfz DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6121d84167620035b567b29c921b42e0a1fe83d4a7ca7fa8abd028a817fc4275/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b9e4381-b8a2-4d0d-ade3-75df4d07810f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/370dbf42-367c-48ac-abb8-86c01bfe2b54/volumes/kubernetes.io~projected/kube-api-access-9nrjm DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/edc57054-ce41-4e4e-97f6-b127bf151d23/volumes/kubernetes.io~projected/kube-api-access-djclq DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e8a2984b-e706-43f8-bd87-dd5bc809c36e/volumes/kubernetes.io~projected/kube-api-access-dd7gp DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:119e068d259449e MacAddress:4a:ab:e9:b1:43:fe Speed:10000 Mtu:8901} {Name:48a8581a2509020 MacAddress:72:fa:57:82:b4:de Speed:10000 Mtu:8901} {Name:4a32f04ab0b6239 MacAddress:ae:ff:ef:ef:22:4b Speed:10000 Mtu:8901} {Name:73444df967788cd MacAddress:a2:20:cf:bf:78:90 Speed:10000 Mtu:8901} {Name:7e840803a5f170f MacAddress:02:67:70:1d:ab:38 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d4:af:df:7b Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:df:2c:d9:0a:f7 Speed:0 Mtu:8901} {Name:cceb6f22eb157fc MacAddress:ba:b1:cf:fb:ba:46 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d4:af:df:7b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:da:3b:f7:ba:2f:08 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8a:39:e0:08:f0:c4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.400498 4573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.400541 4573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.400825 4573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.400843 4573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-145.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.400961 4573 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.400969 4573 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.400984 4573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401005 4573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401086 4573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401241 4573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401344 4573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401377 4573 kubelet.go:493] "Attempting to sync node with API server" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401390 4573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401406 4573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401420 4573 kubelet.go:405] "Adding apiserver pod source" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.401441 4573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.402166 4573 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.402194 4573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:42.406796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.404020 4573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407244 4573 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407256 4573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407361 4573 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407570 4573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407601 4573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407705 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407721 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407730 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407739 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407761 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407773 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407782 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407790 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407800 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407809 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407822 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407836 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407868 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 15:47:42.407892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.407877 4573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 15:47:42.411388 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.411371 4573 server.go:1263] "Started kubelet" Nov 01 15:47:42.411527 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.411481 4573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 15:47:42.411575 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.411548 4573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.411766 4573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.412008 4573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.412835 4573 server.go:317] "Adding debug handlers to kubelet server" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.414246 4573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.414521 4573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.414530 4573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.414546 4573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.414534 4573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.414630 4573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416320 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="370dbf42-367c-48ac-abb8-86c01bfe2b54" volumeName="kubernetes.io/configmap/370dbf42-367c-48ac-abb8-86c01bfe2b54-config-volume" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416348 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ada3c550-b52f-4236-88db-a07ed2c6fdda" volumeName="kubernetes.io/configmap/ada3c550-b52f-4236-88db-a07ed2c6fdda-iptables-alerter-script" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416369 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0563d8c3-dfee-41e5-b9ed-81a71a05c8f7" volumeName="kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-tuned" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416397 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a228ff8-0200-4b6f-8948-9e351926a3b7" volumeName="kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-script-lib" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416408 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a228ff8-0200-4b6f-8948-9e351926a3b7" volumeName="kubernetes.io/secret/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416429 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1" volumeName="kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416440 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8a2984b-e706-43f8-bd87-dd5bc809c36e" volumeName="kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-daemon-config" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416459 4573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416470 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7e5339a-b073-4184-9acc-5a701276ea1c" volumeName="kubernetes.io/configmap/f7e5339a-b073-4184-9acc-5a701276ea1c-konnectivity-ca" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416495 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="370dbf42-367c-48ac-abb8-86c01bfe2b54" volumeName="kubernetes.io/empty-dir/370dbf42-367c-48ac-abb8-86c01bfe2b54-tmp-dir" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416506 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4926376c-9c72-4d94-8505-959c770493a8" volumeName="kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416516 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9e4381-b8a2-4d0d-ade3-75df4d07810f" volumeName="kubernetes.io/secret/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-cert" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416545 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22ea5ba-32e9-4552-802d-d11594d04f29" volumeName="kubernetes.io/projected/a22ea5ba-32e9-4552-802d-d11594d04f29-kube-api-access-7b7q6" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416576 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8620858-3d8f-4222-abf0-0b468a12c8d9" volumeName="kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416587 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8620858-3d8f-4222-abf0-0b468a12c8d9" volumeName="kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416609 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da994e56-8f1d-4f00-893a-670984816adc" volumeName="kubernetes.io/empty-dir/da994e56-8f1d-4f00-893a-670984816adc-data-volume" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416622 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f7e5339a-b073-4184-9acc-5a701276ea1c" volumeName="kubernetes.io/secret/f7e5339a-b073-4184-9acc-5a701276ea1c-agent-certs" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416634 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0563d8c3-dfee-41e5-b9ed-81a71a05c8f7" volumeName="kubernetes.io/projected/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-kube-api-access-blnvk" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416661 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a2d0642-db9c-4bd7-a804-51167fdd62fd" volumeName="kubernetes.io/projected/0a2d0642-db9c-4bd7-a804-51167fdd62fd-kube-api-access-4s446" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416675 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a228ff8-0200-4b6f-8948-9e351926a3b7" volumeName="kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-config" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416699 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a228ff8-0200-4b6f-8948-9e351926a3b7" volumeName="kubernetes.io/projected/4a228ff8-0200-4b6f-8948-9e351926a3b7-kube-api-access-vwkgh" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416723 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff1ca6f5-1f79-49ec-954c-a8bfa9928609" volumeName="kubernetes.io/configmap/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-cabundle" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416738 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0563d8c3-dfee-41e5-b9ed-81a71a05c8f7" volumeName="kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-tmp" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416803 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4926376c-9c72-4d94-8505-959c770493a8" volumeName="kubernetes.io/projected/4926376c-9c72-4d94-8505-959c770493a8-kube-api-access-cksfz" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416834 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a228ff8-0200-4b6f-8948-9e351926a3b7" volumeName="kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-env-overrides" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416867 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4bf6473d-8d38-41da-9d38-d71ddc4957ad" volumeName="kubernetes.io/projected/4bf6473d-8d38-41da-9d38-d71ddc4957ad-kube-api-access-9fvz7" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416891 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a22ea5ba-32e9-4552-802d-d11594d04f29" volumeName="kubernetes.io/empty-dir/a22ea5ba-32e9-4552-802d-d11594d04f29-tmp-dir" seLinuxMountContext="" Nov 01 15:47:42.416950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.416915 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edc57054-ce41-4e4e-97f6-b127bf151d23" volumeName="kubernetes.io/configmap/edc57054-ce41-4e4e-97f6-b127bf151d23-serviceca" seLinuxMountContext="" Nov 01 15:47:42.411846 ip-10-0-141-145 systemd[1]: Started Kubernetes Kubelet. Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418282 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36ef09c4-10c5-4507-98c8-c94ec38f46f7" volumeName="kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418336 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b9e4381-b8a2-4d0d-ade3-75df4d07810f" volumeName="kubernetes.io/projected/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-kube-api-access-brr4h" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418351 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ada3c550-b52f-4236-88db-a07ed2c6fdda" volumeName="kubernetes.io/projected/ada3c550-b52f-4236-88db-a07ed2c6fdda-kube-api-access-6rzcv" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418371 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8a2984b-e706-43f8-bd87-dd5bc809c36e" volumeName="kubernetes.io/projected/e8a2984b-e706-43f8-bd87-dd5bc809c36e-kube-api-access-dd7gp" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418385 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="370dbf42-367c-48ac-abb8-86c01bfe2b54" volumeName="kubernetes.io/projected/370dbf42-367c-48ac-abb8-86c01bfe2b54-kube-api-access-9nrjm" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418411 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4926376c-9c72-4d94-8505-959c770493a8" volumeName="kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418426 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4926376c-9c72-4d94-8505-959c770493a8" volumeName="kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-binary-copy" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418439 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8620858-3d8f-4222-abf0-0b468a12c8d9" volumeName="kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418456 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8a2984b-e706-43f8-bd87-dd5bc809c36e" volumeName="kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cni-binary-copy" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418468 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff1ca6f5-1f79-49ec-954c-a8bfa9928609" volumeName="kubernetes.io/secret/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-key" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418485 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8620858-3d8f-4222-abf0-0b468a12c8d9" volumeName="kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418504 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da994e56-8f1d-4f00-893a-670984816adc" volumeName="kubernetes.io/projected/da994e56-8f1d-4f00-893a-670984816adc-kube-api-access-rn6rl" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418518 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="edc57054-ce41-4e4e-97f6-b127bf151d23" volumeName="kubernetes.io/projected/edc57054-ce41-4e4e-97f6-b127bf151d23-kube-api-access-djclq" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418535 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ff1ca6f5-1f79-49ec-954c-a8bfa9928609" volumeName="kubernetes.io/projected/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-kube-api-access-d4sb5" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418548 4573 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06506c1c-88b0-42e1-ab42-8cc842decd18" volumeName="kubernetes.io/projected/06506c1c-88b0-42e1-ab42-8cc842decd18-kube-api-access-rgr87" seLinuxMountContext="" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418560 4573 reconstruct.go:97] "Volume reconstruction finished" Nov 01 15:47:42.418949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.418567 4573 reconciler.go:29] "Reconciler: start to sync state" Nov 01 15:47:42.419716 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.419074 4573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:42.419881 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.419834 4573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 15:42:03 +0000 UTC" deadline="2028-07-01 01:14:09.883782533 +0000 UTC" Nov 01 15:47:42.419946 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.419882 4573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14577h26m27.463904373s" Nov 01 15:47:42.421578 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.421554 4573 factory.go:55] Registering systemd factory Nov 01 15:47:42.421578 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.421576 4573 factory.go:223] Registration of the systemd container factory successfully Nov 01 15:47:42.422006 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.421977 4573 factory.go:153] Registering CRI-O factory Nov 01 15:47:42.422006 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.421996 4573 factory.go:223] Registration of the crio container factory successfully Nov 01 15:47:42.422134 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.422045 4573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 15:47:42.422134 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.422073 4573 factory.go:103] Registering Raw factory Nov 01 15:47:42.422134 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.422086 4573 manager.go:1199] Started watching for new ooms in manager Nov 01 15:47:42.422416 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.422402 4573 manager.go:319] Starting recovery of all containers Nov 01 15:47:42.424322 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.424301 4573 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 15:47:42.431783 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.431717 4573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 15:47:42.440587 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.440566 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:47:42.440905 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.440885 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:47:42.449348 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.449316 4573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 15:47:42.449348 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.449343 4573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 15:47:42.449487 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.449368 4573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 15:47:42.449487 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:42.449416 4573 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 15:47:42.452803 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.452775 4573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 15:47:42.462280 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.462259 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:47:42.462611 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.462588 4573 generic.go:356] "Generic (PLEG): container finished" podID="4a228ff8-0200-4b6f-8948-9e351926a3b7" containerID="3555c1050014611b1d2fc2394bc085dabcf031d243fb0a937975dcba6eac6a7d" exitCode=1 Nov 01 15:47:42.470834 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.470808 4573 generic.go:356] "Generic (PLEG): container finished" podID="92683d17f4ef39773bb8bc1b1459025a" containerID="4ad8722323c7e02cec9187625b11a5654be046a5cba624f55e7d198548360c5c" exitCode=0 Nov 01 15:47:42.486906 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.486878 4573 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="6c470e973f02041a37c73c286f974933f328f1b7bd376ce051fc9625125ee49f" exitCode=0 Nov 01 15:47:42.486906 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.486905 4573 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="464c956c6a1bfd4c44a6544ef0c1f4f67b9292a462b6a2890bb9393f8aa2b185" exitCode=0 Nov 01 15:47:42.487080 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.486917 4573 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="2fe0756340012a74b2f9fbe77da8206e49c0c316ed235f39f796f98cf6925ef3" exitCode=0 Nov 01 15:47:42.487080 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.486929 4573 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="c4f340dc9fdd41056c9ffe19a9d8be062d61d66e4ef05b7452819293d63fdd0d" exitCode=0 Nov 01 15:47:42.487080 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.486938 4573 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="9a8ed9e695a18ade97eae406710a69e488832bdd25f7e1d1ca10c920e8b35430" exitCode=0 Nov 01 15:47:42.487080 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.486943 4573 generic.go:356] "Generic (PLEG): container finished" podID="4926376c-9c72-4d94-8505-959c770493a8" containerID="e6dcb4972966df4a0425cb6458b7f367b807d4d3f4b4b6f51267ad0c3754f391" exitCode=0 Nov 01 15:47:42.498479 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.498419 4573 manager.go:324] Recovery completed Nov 01 15:47:42.514586 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.514560 4573 cpu_manager.go:226] "Starting" policy="none" Nov 01 15:47:42.514586 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.514576 4573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 15:47:42.514706 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.514596 4573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 15:47:42.514779 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.514764 4573 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 15:47:42.515206 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.514779 4573 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 15:47:42.515206 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.515200 4573 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 15:47:42.515206 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.515207 4573 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 15:47:42.515348 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.515213 4573 policy_none.go:50] "Start" Nov 01 15:47:42.515348 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.515222 4573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 15:47:42.515348 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.515234 4573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 15:47:42.515430 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.515355 4573 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 15:47:42.515430 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.515362 4573 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 15:47:42.515430 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.515368 4573 policy_none.go:44] "Start" Nov 01 15:47:42.519767 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.519729 4573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 15:47:42.524284 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.524270 4573 manager.go:341] "Starting Device Plugin manager" Nov 01 15:47:42.524367 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:42.524299 4573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 15:47:42.524367 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.524310 4573 server.go:87] "Starting device plugin registration server" Nov 01 15:47:42.524610 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.524597 4573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 15:47:42.524677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.524612 4573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 15:47:42.524778 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.524731 4573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 15:47:42.524903 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.524867 4573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 15:47:42.525462 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.525447 4573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 15:47:42.525462 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.525458 4573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 15:47:42.525588 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.525504 4573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T15:47:42.524901379Z","UUID":"cf592eae-e73b-485b-baec-477412667879","Handler":null,"Name":"","Endpoint":""} Nov 01 15:47:42.526302 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:42.526282 4573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 15:47:42.526952 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.526934 4573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 15:47:42.527023 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.526959 4573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 15:47:42.550155 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.550114 4573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal"] Nov 01 15:47:42.550661 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.550621 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"6555262b15c85369c5c728ca31cfdef1","Type":"ContainerStarted","Data":"b31a3718a182f952a7ba978d8e68111da6907fce31f064325591a86bc25694d1"} Nov 01 15:47:42.550697 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.550677 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"6555262b15c85369c5c728ca31cfdef1","Type":"ContainerStarted","Data":"10f5ee1b941f2c53e73e955e1092ccfca4a2a2ff935b6c2763c7e1100b244af0"} Nov 01 15:47:42.550738 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.550700 4573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.550835 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.550700 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"63ae19f081e708ec74a910bf3c54cd46fe5e7ac891d89ac659aa92baf3c221e9"} Nov 01 15:47:42.550881 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.550862 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerDied","Data":"4ad8722323c7e02cec9187625b11a5654be046a5cba624f55e7d198548360c5c"} Nov 01 15:47:42.550914 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.550890 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" event={"ID":"92683d17f4ef39773bb8bc1b1459025a","Type":"ContainerStarted","Data":"ba4251a2aab9ac5f477e533e1061c75fe685c92bc68877976888934f220d0679"} Nov 01 15:47:42.550914 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.550677 4573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.562101 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.562088 4573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:42.562156 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:42.562127 4573 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.562496 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.562486 4573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:42.562529 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:42.562507 4573 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.624827 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.624799 4573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 15:47:42.624948 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.624840 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.624948 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.624871 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.624948 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.624887 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.626082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.626060 4573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 15:47:42.626196 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.626093 4573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 15:47:42.626196 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.626104 4573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-145.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 15:47:42.626196 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.626137 4573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.637904 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.637890 4573 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.637970 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.637948 4573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.726009 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.725974 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.726009 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.726009 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.726186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.726033 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.726186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.726083 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.726186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.726087 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/92683d17f4ef39773bb8bc1b1459025a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"92683d17f4ef39773bb8bc1b1459025a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:42.726186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:42.726087 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6555262b15c85369c5c728ca31cfdef1-config\") pod \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" (UID: \"6555262b15c85369c5c728ca31cfdef1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:43.401978 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.401943 4573 apiserver.go:52] "Watching apiserver" Nov 01 15:47:43.408968 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.408948 4573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 15:47:43.409362 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.409344 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rsjf4","openshift-ovn-kubernetes/ovnkube-node-x4htf","openshift-dns/node-resolver-8fxdr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-p66vf","openshift-network-diagnostics/network-check-target-2xm97","openshift-multus/multus-additional-cni-plugins-bmsrc","openshift-multus/network-metrics-daemon-bdz9j","openshift-dns/dns-default-gx6wg","openshift-ingress-canary/ingress-canary-b7q5z","kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal","kube-system/konnectivity-agent-cvct9","openshift-insights/periodic-gathering-jv28d-6kkx9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw","openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4","openshift-network-operator/iptables-alerter-q52bf","openshift-multus/multus-b6ttz","openshift-insights/insights-runtime-extractor-lsszq","openshift-service-ca/service-ca-5fffc97bdf-6hrlt","kube-system/global-pull-secret-syncer-2g5v9"] Nov 01 15:47:43.414300 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.414268 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:43.414412 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.414300 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 15:47:43.414412 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.414315 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.414412 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.414370 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.414576 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.414498 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-58l9j\"" Nov 01 15:47:43.415072 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415051 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.415163 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415120 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.415163 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415140 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 15:47:43.415279 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415166 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.415279 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415194 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 15:47:43.415279 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415202 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.415279 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415232 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.415279 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415251 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 15:47:43.415279 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415266 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.415717 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415392 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjc9m\"" Nov 01 15:47:43.415717 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415476 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qm9xj\"" Nov 01 15:47:43.415717 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415671 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6x5r5\"" Nov 01 15:47:43.415893 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415778 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 15:47:43.415893 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415811 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-2bqxm\"" Nov 01 15:47:43.415990 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415928 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bwtj5\"" Nov 01 15:47:43.415990 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.415968 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.416089 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.416041 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 15:47:43.416131 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.416123 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.416668 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.416647 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jsz89\"" Nov 01 15:47:43.416928 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.416915 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.417466 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.417446 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.417725 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.417705 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 15:47:43.417871 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.417859 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 15:47:43.417930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.417894 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 15:47:43.418569 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.418552 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.418569 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.418554 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.418782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.418768 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.419224 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.419206 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 15:47:43.420144 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420128 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.420220 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420179 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.420220 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420190 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 01 15:47:43.420220 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420193 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 15:47:43.420220 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420197 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.420380 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420238 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 15:47:43.420380 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420259 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.420380 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420318 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 15:47:43.420768 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.420735 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 15:47:43.421040 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421009 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 15:47:43.421040 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421019 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 15:47:43.421180 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421114 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.421295 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421275 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.421295 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421275 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 15:47:43.421438 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421297 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-28hsn\"" Nov 01 15:47:43.421438 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421299 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 15:47:43.421438 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421316 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5vhjr\"" Nov 01 15:47:43.421438 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421340 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wl7pm\"" Nov 01 15:47:43.421438 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421358 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jbzz7\"" Nov 01 15:47:43.421438 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421438 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9hxd8\"" Nov 01 15:47:43.421694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421518 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.421694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421615 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.421694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421631 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4p8cp\"" Nov 01 15:47:43.421822 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421801 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 15:47:43.421898 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421882 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.421967 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.421882 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-srrqq\"" Nov 01 15:47:43.422074 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.422057 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-ttxgn\"" Nov 01 15:47:43.422158 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.422146 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4c2bg\"" Nov 01 15:47:43.422392 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.422378 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 15:47:43.422474 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.422410 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 01 15:47:43.422474 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.422419 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 01 15:47:43.422999 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.422985 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 01 15:47:43.423072 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.422987 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mtfck\"" Nov 01 15:47:43.423072 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.423060 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 01 15:47:43.423433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.423417 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 01 15:47:43.423518 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.423505 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-rkntg\"" Nov 01 15:47:43.429212 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429198 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-run\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.429264 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429223 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edc57054-ce41-4e4e-97f6-b127bf151d23-serviceca\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:43.429264 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429238 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-conf-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.429264 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429254 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-dbus\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:43.429399 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429304 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-etc-kubernetes\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.429399 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429321 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b7q6\" (UniqueName: \"kubernetes.io/projected/a22ea5ba-32e9-4552-802d-d11594d04f29-kube-api-access-7b7q6\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:43.429484 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429397 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-systemd\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.429484 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429453 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-tmp\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.429562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429487 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-kubelet\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.429562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429530 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-tmp\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.429652 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429532 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-slash\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.429652 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429585 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-netns\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.429652 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429602 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edc57054-ce41-4e4e-97f6-b127bf151d23-serviceca\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:43.429652 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429610 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:43.429652 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429646 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-modprobe-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429675 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-etc-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429704 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-ovn\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429734 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-config\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429781 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-system-cni-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429804 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429821 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-k8s-cni-cncf-io\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429836 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada3c550-b52f-4236-88db-a07ed2c6fdda-host-slash\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429848 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-original-pull-secret\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:43.429873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429857 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-var-lib-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429880 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cksfz\" (UniqueName: \"kubernetes.io/projected/4926376c-9c72-4d94-8505-959c770493a8-kube-api-access-cksfz\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.429905 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-netns\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430010 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-config\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430029 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-socket-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430048 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-registration-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430073 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rzcv\" (UniqueName: \"kubernetes.io/projected/ada3c550-b52f-4236-88db-a07ed2c6fdda-kube-api-access-6rzcv\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430115 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-kubernetes\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430144 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc57054-ce41-4e4e-97f6-b127bf151d23-host\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430175 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-bin\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430193 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-system-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.430227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430208 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430235 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430267 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djclq\" (UniqueName: \"kubernetes.io/projected/edc57054-ce41-4e4e-97f6-b127bf151d23-kube-api-access-djclq\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430293 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovn-node-metrics-cert\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430330 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-etc-selinux\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430362 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-sys\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430387 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-lib-modules\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430421 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s446\" (UniqueName: \"kubernetes.io/projected/0a2d0642-db9c-4bd7-a804-51167fdd62fd-kube-api-access-4s446\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430471 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430497 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430534 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-daemon-config\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430573 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-log-socket\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430602 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwkgh\" (UniqueName: \"kubernetes.io/projected/4a228ff8-0200-4b6f-8948-9e351926a3b7-kube-api-access-vwkgh\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430629 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cni-binary-copy\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.430655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430633 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovn-node-metrics-cert\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430672 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd7gp\" (UniqueName: \"kubernetes.io/projected/e8a2984b-e706-43f8-bd87-dd5bc809c36e-kube-api-access-dd7gp\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430695 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-systemd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430725 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430774 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-var-lib-kubelet\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430799 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-daemon-config\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430830 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-host\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430862 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cni-binary-copy\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430865 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-kubelet-config\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430901 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430931 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-bin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430953 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-hostroot\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.430984 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ada3c550-b52f-4236-88db-a07ed2c6fdda-iptables-alerter-script\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431032 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22ea5ba-32e9-4552-802d-d11594d04f29-tmp-dir\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431033 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431075 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431130 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a22ea5ba-32e9-4552-802d-d11594d04f29-tmp-dir\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431168 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blnvk\" (UniqueName: \"kubernetes.io/projected/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-kube-api-access-blnvk\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.431184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431192 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-env-overrides\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431240 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ada3c550-b52f-4236-88db-a07ed2c6fdda-iptables-alerter-script\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431278 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-script-lib\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431295 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431322 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a22ea5ba-32e9-4552-802d-d11594d04f29-hosts-file\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431408 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysconfig\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431434 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-env-overrides\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431438 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-cnibin\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431474 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cnibin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431484 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4a228ff8-0200-4b6f-8948-9e351926a3b7-ovnkube-script-lib\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431794 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-socket-dir-parent\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431835 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431850 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-multus\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.431942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.431916 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-netd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.434905 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.434883 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.435169 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435155 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-device-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.435216 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435181 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:43.435216 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435196 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-node-log\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.435216 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435212 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.435312 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435226 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-kubelet\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.435312 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435278 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-multus-certs\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.435380 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435325 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-sys-fs\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.435380 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435358 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgr87\" (UniqueName: \"kubernetes.io/projected/06506c1c-88b0-42e1-ab42-8cc842decd18-kube-api-access-rgr87\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.435467 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435401 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-conf\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.435467 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435425 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-tuned\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.435523 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435481 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-systemd-units\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.435523 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435499 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-tuned\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.435613 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.435595 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4926376c-9c72-4d94-8505-959c770493a8-cni-binary-copy\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.438440 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.438420 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b7q6\" (UniqueName: \"kubernetes.io/projected/a22ea5ba-32e9-4552-802d-d11594d04f29-kube-api-access-7b7q6\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:43.439405 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.439387 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s446\" (UniqueName: \"kubernetes.io/projected/0a2d0642-db9c-4bd7-a804-51167fdd62fd-kube-api-access-4s446\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:43.439499 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.439422 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cksfz\" (UniqueName: \"kubernetes.io/projected/4926376c-9c72-4d94-8505-959c770493a8-kube-api-access-cksfz\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.439588 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.439563 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blnvk\" (UniqueName: \"kubernetes.io/projected/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-kube-api-access-blnvk\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.439683 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.439665 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djclq\" (UniqueName: \"kubernetes.io/projected/edc57054-ce41-4e4e-97f6-b127bf151d23-kube-api-access-djclq\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:43.439777 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.439740 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd7gp\" (UniqueName: \"kubernetes.io/projected/e8a2984b-e706-43f8-bd87-dd5bc809c36e-kube-api-access-dd7gp\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.440109 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.440094 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwkgh\" (UniqueName: \"kubernetes.io/projected/4a228ff8-0200-4b6f-8948-9e351926a3b7-kube-api-access-vwkgh\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.440199 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.440180 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rzcv\" (UniqueName: \"kubernetes.io/projected/ada3c550-b52f-4236-88db-a07ed2c6fdda-kube-api-access-6rzcv\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:43.440766 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.440733 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzdnx\" (UniqueName: \"kubernetes.io/projected/36ef09c4-10c5-4507-98c8-c94ec38f46f7-kube-api-access-nzdnx\") pod \"network-check-target-2xm97\" (UID: \"36ef09c4-10c5-4507-98c8-c94ec38f46f7\") " pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:43.440820 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.440807 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgr87\" (UniqueName: \"kubernetes.io/projected/06506c1c-88b0-42e1-ab42-8cc842decd18-kube-api-access-rgr87\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.495530 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.495506 4573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:43.495667 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.495652 4573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:43.506302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.506278 4573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:43.506375 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:43.506331 4573 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:43.506694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.506680 4573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 15:47:43.506725 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:43.506711 4573 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-145.us-east-2.compute.internal" Nov 01 15:47:43.516081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.516065 4573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 15:47:43.535987 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.535965 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.536063 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.535992 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-key\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:43.536063 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536019 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a22ea5ba-32e9-4552-802d-d11594d04f29-hosts-file\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:43.536063 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536033 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysconfig\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.536191 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536091 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysconfig\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.536191 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536119 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a22ea5ba-32e9-4552-802d-d11594d04f29-hosts-file\") pod \"node-resolver-8fxdr\" (UID: \"a22ea5ba-32e9-4552-802d-d11594d04f29\") " pod="openshift-dns/node-resolver-8fxdr" Nov 01 15:47:43.536191 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536166 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.536191 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536165 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-cnibin\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536208 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-cnibin\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536219 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cnibin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536266 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-cnibin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536266 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-socket-dir-parent\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536294 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-multus\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536310 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-cabundle\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536314 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-socket-dir-parent\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536323 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-cert\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536344 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-multus\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536343 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-netd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536272 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-key\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536365 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536374 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-netd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.536379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536382 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/370dbf42-367c-48ac-abb8-86c01bfe2b54-config-volume\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536398 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da994e56-8f1d-4f00-893a-670984816adc-data-volume\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536415 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn6rl\" (UniqueName: \"kubernetes.io/projected/da994e56-8f1d-4f00-893a-670984816adc-kube-api-access-rn6rl\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536418 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536438 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-device-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536486 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-node-log\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536489 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da994e56-8f1d-4f00-893a-670984816adc-data-volume\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536507 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-signing-cabundle\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536509 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-kubelet\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536543 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-cert\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536549 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-kubelet\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536559 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-node-log\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536546 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-device-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536563 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-multus-certs\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536611 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/370dbf42-367c-48ac-abb8-86c01bfe2b54-config-volume\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536611 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-sys-fs\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536644 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-multus-certs\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536669 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-conf\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536689 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-systemd-units\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536703 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-run\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536671 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-sys-fs\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536718 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-conf-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536742 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nrjm\" (UniqueName: \"kubernetes.io/projected/370dbf42-367c-48ac-abb8-86c01bfe2b54-kube-api-access-9nrjm\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536765 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-systemd-units\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536787 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-conf-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536795 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-sysctl-conf\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536788 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-run\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536785 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-dbus\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536828 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-dbus\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536837 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-etc-kubernetes\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536865 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536889 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-systemd\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536914 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-kubelet\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536891 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-etc-kubernetes\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536935 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-slash\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536948 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-systemd\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536955 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536966 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-netns\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536977 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-slash\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.536983 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-kubelet\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537015 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-modprobe-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537042 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-etc-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.537042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537062 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-netns\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537091 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-modprobe-d\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537107 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-ovn\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537120 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-etc-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537151 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-system-cni-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537189 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-ovn\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537189 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-system-cni-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537202 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537221 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-k8s-cni-cncf-io\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537237 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537278 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-k8s-cni-cncf-io\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537260 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada3c550-b52f-4236-88db-a07ed2c6fdda-host-slash\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537293 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-multus-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537322 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-var-lib-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537324 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ada3c550-b52f-4236-88db-a07ed2c6fdda-host-slash\") pod \"iptables-alerter-q52bf\" (UID: \"ada3c550-b52f-4236-88db-a07ed2c6fdda\") " pod="openshift-network-operator/iptables-alerter-q52bf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537354 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-netns\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537376 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-run-netns\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537358 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-var-lib-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537382 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537482 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da994e56-8f1d-4f00-893a-670984816adc-crio-socket\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537503 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-socket-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537512 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537521 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-registration-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537559 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-socket-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537565 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-registration-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537581 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-kubernetes\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537598 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc57054-ce41-4e4e-97f6-b127bf151d23-host\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537611 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-bin\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537550 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537628 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-system-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537652 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvz7\" (UniqueName: \"kubernetes.io/projected/4bf6473d-8d38-41da-9d38-d71ddc4957ad-kube-api-access-9fvz7\") pod \"migrator-66df5d496b-fq2w4\" (UID: \"4bf6473d-8d38-41da-9d38-d71ddc4957ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537663 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-cni-bin\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537672 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-etc-kubernetes\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537675 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537698 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-system-cni-dir\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537702 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc57054-ce41-4e4e-97f6-b127bf151d23-host\") pod \"node-ca-rsjf4\" (UID: \"edc57054-ce41-4e4e-97f6-b127bf151d23\") " pod="openshift-image-registry/node-ca-rsjf4" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537784 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537788 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537809 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brr4h\" (UniqueName: \"kubernetes.io/projected/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-kube-api-access-brr4h\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537832 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537848 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370dbf42-367c-48ac-abb8-86c01bfe2b54-tmp-dir\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537863 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537880 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-etc-selinux\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537896 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-sys\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537922 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-lib-modules\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537947 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537963 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.537992 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/370dbf42-367c-48ac-abb8-86c01bfe2b54-tmp-dir\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538001 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-openvswitch\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538009 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-sys\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538088 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4sb5\" (UniqueName: \"kubernetes.io/projected/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-kube-api-access-d4sb5\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538114 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538119 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-lib-modules\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538123 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-log-socket\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538161 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-log-socket\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538155 4573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538193 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7e5339a-b073-4184-9acc-5a701276ea1c-agent-certs\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538220 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7e5339a-b073-4184-9acc-5a701276ea1c-konnectivity-ca\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538247 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538273 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-systemd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538297 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-var-lib-kubelet\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538320 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-host\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538320 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/06506c1c-88b0-42e1-ab42-8cc842decd18-etc-selinux\") pod \"aws-ebs-csi-driver-node-nj7nw\" (UID: \"06506c1c-88b0-42e1-ab42-8cc842decd18\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nj7nw" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538393 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-kubelet-config\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538407 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f7e5339a-b073-4184-9acc-5a701276ea1c-konnectivity-ca\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538422 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538445 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-bin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538454 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-host\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538456 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4a228ff8-0200-4b6f-8948-9e351926a3b7-run-systemd\") pod \"ovnkube-node-x4htf\" (UID: \"4a228ff8-0200-4b6f-8948-9e351926a3b7\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:43.538433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538466 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-hostroot\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.540344 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538495 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4926376c-9c72-4d94-8505-959c770493a8-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bmsrc\" (UID: \"4926376c-9c72-4d94-8505-959c770493a8\") " pod="openshift-multus/multus-additional-cni-plugins-bmsrc" Nov 01 15:47:43.540344 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538505 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1-kubelet-config\") pod \"global-pull-secret-syncer-2g5v9\" (UID: \"81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1\") " pod="kube-system/global-pull-secret-syncer-2g5v9" Nov 01 15:47:43.540344 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538492 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.540344 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538540 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-hostroot\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.540344 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538579 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e8a2984b-e706-43f8-bd87-dd5bc809c36e-host-var-lib-cni-bin\") pod \"multus-b6ttz\" (UID: \"e8a2984b-e706-43f8-bd87-dd5bc809c36e\") " pod="openshift-multus/multus-b6ttz" Nov 01 15:47:43.540344 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538626 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0563d8c3-dfee-41e5-b9ed-81a71a05c8f7-var-lib-kubelet\") pod \"tuned-p66vf\" (UID: \"0563d8c3-dfee-41e5-b9ed-81a71a05c8f7\") " pod="openshift-cluster-node-tuning-operator/tuned-p66vf" Nov 01 15:47:43.540344 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.538672 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f7e5339a-b073-4184-9acc-5a701276ea1c-agent-certs\") pod \"konnectivity-agent-cvct9\" (UID: \"f7e5339a-b073-4184-9acc-5a701276ea1c\") " pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:43.540344 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.540250 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a2d0642-db9c-4bd7-a804-51167fdd62fd-metrics-certs\") pod \"network-metrics-daemon-bdz9j\" (UID: \"0a2d0642-db9c-4bd7-a804-51167fdd62fd\") " pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:43.542155 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.542139 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nrjm\" (UniqueName: \"kubernetes.io/projected/370dbf42-367c-48ac-abb8-86c01bfe2b54-kube-api-access-9nrjm\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.542269 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.542252 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn6rl\" (UniqueName: \"kubernetes.io/projected/da994e56-8f1d-4f00-893a-670984816adc-kube-api-access-rn6rl\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.543684 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.543657 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brr4h\" (UniqueName: \"kubernetes.io/projected/8b9e4381-b8a2-4d0d-ade3-75df4d07810f-kube-api-access-brr4h\") pod \"ingress-canary-b7q5z\" (UID: \"8b9e4381-b8a2-4d0d-ade3-75df4d07810f\") " pod="openshift-ingress-canary/ingress-canary-b7q5z" Nov 01 15:47:43.543964 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.543943 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvz7\" (UniqueName: \"kubernetes.io/projected/4bf6473d-8d38-41da-9d38-d71ddc4957ad-kube-api-access-9fvz7\") pod \"migrator-66df5d496b-fq2w4\" (UID: \"4bf6473d-8d38-41da-9d38-d71ddc4957ad\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fq2w4" Nov 01 15:47:43.544024 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.543975 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4sb5\" (UniqueName: \"kubernetes.io/projected/ff1ca6f5-1f79-49ec-954c-a8bfa9928609-kube-api-access-d4sb5\") pod \"service-ca-5fffc97bdf-6hrlt\" (UID: \"ff1ca6f5-1f79-49ec-954c-a8bfa9928609\") " pod="openshift-service-ca/service-ca-5fffc97bdf-6hrlt" Nov 01 15:47:43.544127 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.544111 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r\") pod \"periodic-gathering-jv28d-6kkx9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:47:43.639028 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.638992 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da994e56-8f1d-4f00-893a-670984816adc-crio-socket\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.639028 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.639034 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.639238 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.639054 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:43.639238 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.639075 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.639238 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.639135 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da994e56-8f1d-4f00-893a-670984816adc-crio-socket\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.639238 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:43.639165 4573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:43.639238 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:43.639240 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.13922334 +0000 UTC m=+1.842389267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:43.639427 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:43.639166 4573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:43.639427 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:43.639329 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls podName:370dbf42-367c-48ac-abb8-86c01bfe2b54 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:44.139311802 +0000 UTC m=+1.842477734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls") pod "dns-default-gx6wg" (UID: "370dbf42-367c-48ac-abb8-86c01bfe2b54") : secret "dns-default-metrics-tls" not found Nov 01 15:47:43.639617 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.639600 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da994e56-8f1d-4f00-893a-670984816adc-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:43.717149 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.717069 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bdz9j" Nov 01 15:47:43.836549 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:43.836521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:44.024164 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:44.024081 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bdz9j"] Nov 01 15:47:44.027924 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:44.027880 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2d0642_db9c_4bd7_a804_51167fdd62fd.slice/crio-deec3e3fb5cbaa82ad42f3f86e5535ec99fc8b96b32bcb245e9bb95057d61ffd WatchSource:0}: Error finding container deec3e3fb5cbaa82ad42f3f86e5535ec99fc8b96b32bcb245e9bb95057d61ffd: Status 404 returned error can't find the container with id deec3e3fb5cbaa82ad42f3f86e5535ec99fc8b96b32bcb245e9bb95057d61ffd Nov 01 15:47:44.029884 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:44.029866 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:47:44.058992 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:44.058967 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:44.146247 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:44.146209 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:44.146247 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:44.146252 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:44.146424 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:44.146353 4573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 01 15:47:44.146424 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:44.146355 4573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:44.146424 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:44.146403 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls podName:370dbf42-367c-48ac-abb8-86c01bfe2b54 nodeName:}" failed. No retries permitted until 2026-11-01 15:47:45.146390759 +0000 UTC m=+2.849556673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls") pod "dns-default-gx6wg" (UID: "370dbf42-367c-48ac-abb8-86c01bfe2b54") : secret "dns-default-metrics-tls" not found Nov 01 15:47:44.146510 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:44.146425 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:45.14641243 +0000 UTC m=+2.849578345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:44.429382 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:44.429352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:44.498469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:44.498434 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdz9j" event={"ID":"0a2d0642-db9c-4bd7-a804-51167fdd62fd","Type":"ContainerStarted","Data":"deec3e3fb5cbaa82ad42f3f86e5535ec99fc8b96b32bcb245e9bb95057d61ffd"} Nov 01 15:47:44.630571 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:44.630504 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:45.020351 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.020330 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8fxdr_a22ea5ba-32e9-4552-802d-d11594d04f29/dns-node-resolver/0.log" Nov 01 15:47:45.030023 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.029998 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:45.154716 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.154692 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:45.154844 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.154726 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:45.154889 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:45.154846 4573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:45.154922 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:45.154904 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:47.154888871 +0000 UTC m=+4.858054805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:45.156811 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.156790 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/370dbf42-367c-48ac-abb8-86c01bfe2b54-metrics-tls\") pod \"dns-default-gx6wg\" (UID: \"370dbf42-367c-48ac-abb8-86c01bfe2b54\") " pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:45.202663 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.202635 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:47:45.217613 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.217586 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:45.233247 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.233218 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:45.485269 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.485231 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gx6wg"] Nov 01 15:47:45.489043 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:45.489015 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370dbf42_367c_48ac_abb8_86c01bfe2b54.slice/crio-084e6c87a75891e3f103fa36d119cc9321a20b7f273ca750834d9e23cd49865c WatchSource:0}: Error finding container 084e6c87a75891e3f103fa36d119cc9321a20b7f273ca750834d9e23cd49865c: Status 404 returned error can't find the container with id 084e6c87a75891e3f103fa36d119cc9321a20b7f273ca750834d9e23cd49865c Nov 01 15:47:45.503835 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.503808 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdz9j" event={"ID":"0a2d0642-db9c-4bd7-a804-51167fdd62fd","Type":"ContainerStarted","Data":"fac3065df66757fac275e1e23871278ef94e6a107238de45b7fd2d5091c5de52"} Nov 01 15:47:45.503919 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.503861 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bdz9j" event={"ID":"0a2d0642-db9c-4bd7-a804-51167fdd62fd","Type":"ContainerStarted","Data":"b0bc8ffbc3bcdd9cae7f02401f8f28a4f58e5cf553d08d1ffab8234aad5b565e"} Nov 01 15:47:45.504992 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.504971 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx6wg" event={"ID":"370dbf42-367c-48ac-abb8-86c01bfe2b54","Type":"ContainerStarted","Data":"084e6c87a75891e3f103fa36d119cc9321a20b7f273ca750834d9e23cd49865c"} Nov 01 15:47:45.517172 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.517115 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:45.636720 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.636689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:45.682320 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.682298 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cvct9" Nov 01 15:47:45.734923 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.734898 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:45.836405 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.836334 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:45.935746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:45.935718 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:46.420954 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:46.420925 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rsjf4_edc57054-ce41-4e4e-97f6-b127bf151d23/node-ca/0.log" Nov 01 15:47:46.430230 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:46.430198 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:46.530781 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:46.530734 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:46.820970 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:46.820896 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b7q5z_8b9e4381-b8a2-4d0d-ade3-75df4d07810f/serve-healthcheck-canary/0.log" Nov 01 15:47:46.830313 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:46.830282 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:46.932183 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:46.932150 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:47.031779 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.031097 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:47.131658 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.131600 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:47.169795 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.169770 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:47.169908 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:47.169895 4573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 01 15:47:47.169960 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:47:47.169956 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls podName:da994e56-8f1d-4f00-893a-670984816adc nodeName:}" failed. No retries permitted until 2026-11-01 15:47:51.169941779 +0000 UTC m=+8.873107707 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls") pod "insights-runtime-extractor-lsszq" (UID: "da994e56-8f1d-4f00-893a-670984816adc") : secret "insights-runtime-extractor-tls" not found Nov 01 15:47:47.221181 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.221150 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fq2w4_4bf6473d-8d38-41da-9d38-d71ddc4957ad/migrator/0.log" Nov 01 15:47:47.235004 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.234980 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:47.512926 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.512890 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx6wg" event={"ID":"370dbf42-367c-48ac-abb8-86c01bfe2b54","Type":"ContainerStarted","Data":"8c13e1544832cf72e2a2072e0590f415695da4b9e7e191969070709b016146c8"} Nov 01 15:47:47.513117 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.512937 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx6wg" event={"ID":"370dbf42-367c-48ac-abb8-86c01bfe2b54","Type":"ContainerStarted","Data":"cba8711dff2b4db006fc97edb98941edce8e3704a6fa22b1a269e89b38a91288"} Nov 01 15:47:47.513117 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.513037 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:47.630264 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:47.630230 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:48.429358 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:48.429327 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:48.834407 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:48.834330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:49.036488 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:49.036459 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:49.232033 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:49.232000 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:49.628685 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:49.628647 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:50.081519 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:50.081445 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:50.309190 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:50.309158 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:47:50.428850 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:50.428822 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:50.522278 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:50.521902 4573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 15:47:50.833531 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:50.833454 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:50.933347 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:50.933283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:51.033428 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.033403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:51.195944 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.195915 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:51.198211 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.198188 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da994e56-8f1d-4f00-893a-670984816adc-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-lsszq\" (UID: \"da994e56-8f1d-4f00-893a-670984816adc\") " pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:51.219060 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.219036 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-lsszq" Nov 01 15:47:51.231030 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.231009 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:51.429002 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.428969 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:51.484845 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.484770 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-lsszq"] Nov 01 15:47:51.488251 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:47:51.488216 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda994e56_8f1d_4f00_893a_670984816adc.slice/crio-4381fc6fc321410ed3b80003c9b5e6962eb0fd1f562cf6498288d5e103479771 WatchSource:0}: Error finding container 4381fc6fc321410ed3b80003c9b5e6962eb0fd1f562cf6498288d5e103479771: Status 404 returned error can't find the container with id 4381fc6fc321410ed3b80003c9b5e6962eb0fd1f562cf6498288d5e103479771 Nov 01 15:47:51.526484 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.526461 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lsszq" event={"ID":"da994e56-8f1d-4f00-893a-670984816adc","Type":"ContainerStarted","Data":"4381fc6fc321410ed3b80003c9b5e6962eb0fd1f562cf6498288d5e103479771"} Nov 01 15:47:51.722691 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.722657 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:51.789778 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:51.789699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:52.531043 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:52.531007 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lsszq" event={"ID":"da994e56-8f1d-4f00-893a-670984816adc","Type":"ContainerStarted","Data":"310f844350cff4b4df5c6a95497fe764e016ae41401a9277953cf7b562bb4eda"} Nov 01 15:47:52.531465 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:52.531078 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lsszq" event={"ID":"da994e56-8f1d-4f00-893a-670984816adc","Type":"ContainerStarted","Data":"dcb86b5cd0146dfa0c0494ff491be0f372160843f6cf966c564632f5d919398d"} Nov 01 15:47:54.540314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:54.540277 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-lsszq" event={"ID":"da994e56-8f1d-4f00-893a-670984816adc","Type":"ContainerStarted","Data":"f774f95cfea091b438cb32861c29f15f785b6791b62c74590a81de208a0cf89d"} Nov 01 15:47:56.847697 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:56.847663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:56.941147 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:56.941090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:47:57.518398 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:57.518368 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gx6wg" Nov 01 15:47:59.935495 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:47:59.935462 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:00.196061 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:00.195986 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:02.090165 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:02.090127 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:04.143247 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.143213 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:04.243025 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.242996 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:04.346252 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.346219 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:04.393194 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.393158 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6dxbr"] Nov 01 15:48:04.396526 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.396474 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.440450 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.440421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:04.467222 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.467196 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 15:48:04.467570 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.467554 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-z6wzh\"" Nov 01 15:48:04.467670 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.467656 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 15:48:04.482832 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.482809 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-wtmp\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.482913 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.482840 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/312b0901-ccbf-46ea-952e-2c36151d6122-metrics-client-ca\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.482913 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.482857 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-sys\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.482913 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.482874 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.483058 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.483039 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-accelerators-collector-config\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.483106 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.483075 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-root\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.483163 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.483146 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-textfile\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.483215 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.483188 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-tls\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.483268 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.483252 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzrp\" (UniqueName: \"kubernetes.io/projected/312b0901-ccbf-46ea-952e-2c36151d6122-kube-api-access-nxzrp\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.501792 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.501767 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 15:48:04.501792 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.501777 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 15:48:04.501967 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.501806 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 15:48:04.501967 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.501874 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 15:48:04.584030 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584011 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/312b0901-ccbf-46ea-952e-2c36151d6122-metrics-client-ca\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584132 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584038 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-sys\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584132 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584054 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584132 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584090 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-accelerators-collector-config\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584132 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584119 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-root\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584337 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584144 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-sys\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584337 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584156 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-textfile\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584337 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584189 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-tls\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584337 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584217 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzrp\" (UniqueName: \"kubernetes.io/projected/312b0901-ccbf-46ea-952e-2c36151d6122-kube-api-access-nxzrp\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584337 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584244 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-wtmp\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584337 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584248 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-root\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584616 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584392 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-wtmp\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584673 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584654 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/312b0901-ccbf-46ea-952e-2c36151d6122-metrics-client-ca\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584733 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584696 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-textfile\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.584843 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.584825 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-accelerators-collector-config\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.586550 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.586532 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.586599 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.586584 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/312b0901-ccbf-46ea-952e-2c36151d6122-node-exporter-tls\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.592122 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.592105 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzrp\" (UniqueName: \"kubernetes.io/projected/312b0901-ccbf-46ea-952e-2c36151d6122-kube-api-access-nxzrp\") pod \"node-exporter-6dxbr\" (UID: \"312b0901-ccbf-46ea-952e-2c36151d6122\") " pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.711657 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.711584 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6dxbr" Nov 01 15:48:04.723403 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:48:04.723379 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312b0901_ccbf_46ea_952e_2c36151d6122.slice/crio-3ee355c38718dd81e7de481e747a4c24ab4131eb95bf7298c1ca98ce34f39429 WatchSource:0}: Error finding container 3ee355c38718dd81e7de481e747a4c24ab4131eb95bf7298c1ca98ce34f39429: Status 404 returned error can't find the container with id 3ee355c38718dd81e7de481e747a4c24ab4131eb95bf7298c1ca98ce34f39429 Nov 01 15:48:04.726007 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:04.725980 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:05.013816 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.013725 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:05.366331 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.366268 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:05.444211 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.444165 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:05.452765 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.452715 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:05.464862 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.464810 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:05.465219 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.465199 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.468514 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.468492 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 15:48:05.471650 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.471629 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 15:48:05.471746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.471682 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 15:48:05.471746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.471699 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 15:48:05.471746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.471685 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 15:48:05.474886 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.474868 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 15:48:05.578766 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.573993 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 15:48:05.578766 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.574193 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 15:48:05.578766 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.574489 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 15:48:05.578766 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.575597 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q2zk9\"" Nov 01 15:48:05.578766 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.577805 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6dxbr" event={"ID":"312b0901-ccbf-46ea-952e-2c36151d6122","Type":"ContainerStarted","Data":"3ee355c38718dd81e7de481e747a4c24ab4131eb95bf7298c1ca98ce34f39429"} Nov 01 15:48:05.591466 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591440 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591584 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591482 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591584 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591575 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nvn8\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-kube-api-access-4nvn8\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591671 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591614 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591671 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591646 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591769 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591672 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591810 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591759 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591845 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591830 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591899 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591849 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-web-config\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.591899 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591865 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-out\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.592003 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.591985 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.592039 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.592015 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.592072 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.592040 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692612 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692534 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692612 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692575 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692612 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692600 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nvn8\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-kube-api-access-4nvn8\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692627 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692658 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692680 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692703 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692732 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692774 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-web-config\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692805 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-out\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.692916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692910 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.693322 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692931 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.693322 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.692955 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.693322 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:05.693052 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle podName:e4f0fea5-3aed-40c6-bf06-3215c3ba40ce nodeName:}" failed. No retries permitted until 2026-11-01 15:48:06.193022903 +0000 UTC m=+23.896188830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce") : configmap references non-existent config key: ca-bundle.crt Nov 01 15:48:05.694875 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.693948 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.694875 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.694596 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.695731 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.695704 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.695850 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.695762 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.695918 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.695869 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.696004 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.695973 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-web-config\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.696117 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.695973 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.696466 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.696442 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.696628 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.696613 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-volume\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.697545 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.697526 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-out\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.697678 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.697661 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.700469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.700447 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nvn8\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-kube-api-access-4nvn8\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:05.786721 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:05.786689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:06.197779 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:06.197729 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:06.198458 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:06.198438 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:06.397937 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:06.397907 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:06.582467 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:06.582430 4573 generic.go:356] "Generic (PLEG): container finished" podID="312b0901-ccbf-46ea-952e-2c36151d6122" containerID="a70d5354621594f2e95cabbacf403c8926a8c3b6de115217c71b0a85ef1126dd" exitCode=0 Nov 01 15:48:06.582630 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:06.582482 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6dxbr" event={"ID":"312b0901-ccbf-46ea-952e-2c36151d6122","Type":"ContainerDied","Data":"a70d5354621594f2e95cabbacf403c8926a8c3b6de115217c71b0a85ef1126dd"} Nov 01 15:48:06.677294 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:06.677268 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:06.680212 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:48:06.680188 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f0fea5_3aed_40c6_bf06_3215c3ba40ce.slice/crio-4b78debb3096b7651249df265b4772d42430761756734ebf1b511724b24cfa09 WatchSource:0}: Error finding container 4b78debb3096b7651249df265b4772d42430761756734ebf1b511724b24cfa09: Status 404 returned error can't find the container with id 4b78debb3096b7651249df265b4772d42430761756734ebf1b511724b24cfa09 Nov 01 15:48:06.724871 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:06.724809 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:06.828935 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:06.828905 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:07.050931 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.050861 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:07.168635 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.168594 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:07.324449 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.324378 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:07.424635 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.424595 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:07.491813 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.491775 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:07.586437 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.586357 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerStarted","Data":"4b78debb3096b7651249df265b4772d42430761756734ebf1b511724b24cfa09"} Nov 01 15:48:07.588360 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.588333 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6dxbr" event={"ID":"312b0901-ccbf-46ea-952e-2c36151d6122","Type":"ContainerStarted","Data":"bf402f85ca309ca2a98d00d1e3d43723e2f62b386f8855d8166b36e19b87b5f3"} Nov 01 15:48:07.588477 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.588376 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6dxbr" event={"ID":"312b0901-ccbf-46ea-952e-2c36151d6122","Type":"ContainerStarted","Data":"0dc0dda6a5e590f637776ebaa18c80965ae5a7364ff59911b9593b220f15e498"} Nov 01 15:48:07.607802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.607724 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6dxbr" podStartSLOduration=2.5295840910000003 podStartE2EDuration="3.607708145s" totalImagesPullingTime="1.016025937s" totalInitContainerRuntime="62.098117ms" isStatefulPod=true podCreationTimestamp="2026-11-01 15:48:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:07.606624714 +0000 UTC m=+25.309790654" watchObservedRunningTime="2026-11-01 15:48:07.607708145 +0000 UTC m=+25.310874083" Nov 01 15:48:07.665475 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.665450 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:07.767893 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:07.767865 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:08.592856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.592821 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerID="5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d" exitCode=0 Nov 01 15:48:08.593280 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.592906 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerDied","Data":"5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d"} Nov 01 15:48:08.761127 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.761091 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-855bfdd46d-trf7z"] Nov 01 15:48:08.764107 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.764091 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:08.767312 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.767289 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 15:48:08.767425 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.767325 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Nov 01 15:48:08.772288 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.772268 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-855bfdd46d-trf7z"] Nov 01 15:48:08.809972 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.809938 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Nov 01 15:48:08.810104 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.809989 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Nov 01 15:48:08.810104 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.810037 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-a8g2pjftlcu2b\"" Nov 01 15:48:08.810104 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.810086 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-78zr2\"" Nov 01 15:48:08.918822 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.918791 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-secret-metrics-server-tls\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:08.918989 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.918826 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsbwx\" (UniqueName: \"kubernetes.io/projected/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-kube-api-access-hsbwx\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:08.918989 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.918854 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:08.918989 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.918872 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-secret-metrics-server-client-certs\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:08.918989 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.918966 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-client-ca-bundle\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:08.919170 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.919030 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-metrics-server-audit-profiles\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:08.919170 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:08.919047 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-audit-log\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020066 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020038 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020066 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020069 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-secret-metrics-server-client-certs\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020274 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020085 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-client-ca-bundle\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020274 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020107 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-metrics-server-audit-profiles\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020274 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020124 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-audit-log\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020274 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020172 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-secret-metrics-server-tls\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020274 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020193 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsbwx\" (UniqueName: \"kubernetes.io/projected/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-kube-api-access-hsbwx\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020629 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-audit-log\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.020984 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.020959 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.021122 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.021105 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-metrics-server-audit-profiles\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.022638 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.022621 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-secret-metrics-server-client-certs\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.023025 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.023011 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-client-ca-bundle\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.023188 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.023167 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-secret-metrics-server-tls\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.028263 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.028233 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsbwx\" (UniqueName: \"kubernetes.io/projected/f008bfb6-889a-4049-bbb4-a0eb73f9cba0-kube-api-access-hsbwx\") pod \"metrics-server-855bfdd46d-trf7z\" (UID: \"f008bfb6-889a-4049-bbb4-a0eb73f9cba0\") " pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.075255 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.075231 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:09.154924 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.154883 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm"] Nov 01 15:48:09.158601 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.158573 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:09.165962 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.165889 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm"] Nov 01 15:48:09.172044 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.168983 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w4gmw\"" Nov 01 15:48:09.172044 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.170459 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:09.172044 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.170561 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Nov 01 15:48:09.221447 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.221412 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrdt\" (UniqueName: \"kubernetes.io/projected/565fe20a-c405-4502-8bf1-3bafaa7dc4c3-kube-api-access-vsrdt\") pod \"monitoring-plugin-77f4f76594-zzpzm\" (UID: \"565fe20a-c405-4502-8bf1-3bafaa7dc4c3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:09.221624 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.221552 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/565fe20a-c405-4502-8bf1-3bafaa7dc4c3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zzpzm\" (UID: \"565fe20a-c405-4502-8bf1-3bafaa7dc4c3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:09.239082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.239048 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4htf" Nov 01 15:48:09.267511 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.267474 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:09.322112 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.322080 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrdt\" (UniqueName: \"kubernetes.io/projected/565fe20a-c405-4502-8bf1-3bafaa7dc4c3-kube-api-access-vsrdt\") pod \"monitoring-plugin-77f4f76594-zzpzm\" (UID: \"565fe20a-c405-4502-8bf1-3bafaa7dc4c3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:09.322284 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.322221 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/565fe20a-c405-4502-8bf1-3bafaa7dc4c3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zzpzm\" (UID: \"565fe20a-c405-4502-8bf1-3bafaa7dc4c3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:09.324864 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.324838 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/565fe20a-c405-4502-8bf1-3bafaa7dc4c3-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-zzpzm\" (UID: \"565fe20a-c405-4502-8bf1-3bafaa7dc4c3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:09.330107 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.330084 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrdt\" (UniqueName: \"kubernetes.io/projected/565fe20a-c405-4502-8bf1-3bafaa7dc4c3-kube-api-access-vsrdt\") pod \"monitoring-plugin-77f4f76594-zzpzm\" (UID: \"565fe20a-c405-4502-8bf1-3bafaa7dc4c3\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:09.357668 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.357641 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-855bfdd46d-trf7z"] Nov 01 15:48:09.360797 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:48:09.360773 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf008bfb6_889a_4049_bbb4_a0eb73f9cba0.slice/crio-f05dd5746952cf9319339e093dee940f97c4c245eace3a0f12adcd32c35966c3 WatchSource:0}: Error finding container f05dd5746952cf9319339e093dee940f97c4c245eace3a0f12adcd32c35966c3: Status 404 returned error can't find the container with id f05dd5746952cf9319339e093dee940f97c4c245eace3a0f12adcd32c35966c3 Nov 01 15:48:09.475523 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.475443 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:09.597296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.597266 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" event={"ID":"f008bfb6-889a-4049-bbb4-a0eb73f9cba0","Type":"ContainerStarted","Data":"f05dd5746952cf9319339e093dee940f97c4c245eace3a0f12adcd32c35966c3"} Nov 01 15:48:09.751451 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:09.751375 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm"] Nov 01 15:48:09.754800 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:48:09.754768 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod565fe20a_c405_4502_8bf1_3bafaa7dc4c3.slice/crio-c0f66fba0039602a73e8fd509d45239185e50882c6833dfca18518648eaf814d WatchSource:0}: Error finding container c0f66fba0039602a73e8fd509d45239185e50882c6833dfca18518648eaf814d: Status 404 returned error can't find the container with id c0f66fba0039602a73e8fd509d45239185e50882c6833dfca18518648eaf814d Nov 01 15:48:10.564464 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.564390 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:48:10.568892 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.568867 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.574261 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.574799 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.575118 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hx8tz\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.575241 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.575287 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.575295 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.576105 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.577825 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.577891 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.577952 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.578221 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ausg30f60sle\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.578318 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.578460 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.578564 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 15:48:10.579787 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.578597 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 15:48:10.587519 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.585382 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:48:10.587519 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.587018 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 15:48:10.589468 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.589445 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:10.607649 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.607625 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerStarted","Data":"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc"} Nov 01 15:48:10.608030 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.607716 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerStarted","Data":"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4"} Nov 01 15:48:10.608030 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.607781 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerStarted","Data":"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282"} Nov 01 15:48:10.608030 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.607818 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerStarted","Data":"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413"} Nov 01 15:48:10.608030 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.607846 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerStarted","Data":"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8"} Nov 01 15:48:10.609483 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.609447 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" event={"ID":"565fe20a-c405-4502-8bf1-3bafaa7dc4c3","Type":"ContainerStarted","Data":"c0f66fba0039602a73e8fd509d45239185e50882c6833dfca18518648eaf814d"} Nov 01 15:48:10.632069 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632031 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632109 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632172 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632262 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632204 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632262 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632222 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632262 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632236 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29jt5\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-kube-api-access-29jt5\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632262 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632258 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632420 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632401 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632463 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632432 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632515 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632466 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632515 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632492 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632515 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632512 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632660 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632531 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632660 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632551 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632660 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632582 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-config\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632660 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632596 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632660 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632611 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632660 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632630 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632968 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632647 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.632968 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.632695 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.733468 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733440 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.733468 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733472 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.733695 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733492 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.733695 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733515 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.733695 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733573 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.733695 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733616 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-config\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.733931 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733906 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734050 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733945 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734050 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.733982 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734050 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734008 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734050 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734039 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734067 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734098 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734137 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734164 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734189 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734212 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29jt5\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-kube-api-access-29jt5\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734241 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734277 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734659 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734301 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.734715 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.734697 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.736047 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.736021 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.736484 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.736460 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.737269 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.737245 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.738701 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.738340 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.739578 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.739096 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.739949 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.739927 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.740302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.740277 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-config\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.740302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.740292 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.740503 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.740480 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.740570 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.740551 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.740769 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.740733 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.740969 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.740947 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.741471 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.741445 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.741919 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.741896 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.742195 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.742173 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.742786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.742764 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.743175 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.743144 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.744256 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.744233 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.746827 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.746806 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29jt5\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-kube-api-access-29jt5\") pod \"prometheus-k8s-0\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:10.905904 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:10.905872 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:11.223813 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:11.223722 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:48:11.227936 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:48:11.227906 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff186e3e_3101_466a_b675_a6cfe3d61410.slice/crio-70a401034168bfd0f962d5bed8d380cfc4c7285934f4719027140b0ec685309a WatchSource:0}: Error finding container 70a401034168bfd0f962d5bed8d380cfc4c7285934f4719027140b0ec685309a: Status 404 returned error can't find the container with id 70a401034168bfd0f962d5bed8d380cfc4c7285934f4719027140b0ec685309a Nov 01 15:48:11.613653 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:11.613619 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerID="a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1" exitCode=0 Nov 01 15:48:11.614044 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:11.613705 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerDied","Data":"a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1"} Nov 01 15:48:11.614044 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:11.613797 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerStarted","Data":"70a401034168bfd0f962d5bed8d380cfc4c7285934f4719027140b0ec685309a"} Nov 01 15:48:12.619346 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:12.619130 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" event={"ID":"565fe20a-c405-4502-8bf1-3bafaa7dc4c3","Type":"ContainerStarted","Data":"b91c41b44bc93d49ecb7c492e6dbee85b9eb99def95cf44d4432f9d6ce90452d"} Nov 01 15:48:12.619346 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:12.619327 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:12.621305 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:12.621220 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" event={"ID":"f008bfb6-889a-4049-bbb4-a0eb73f9cba0","Type":"ContainerStarted","Data":"2622822a551394b09524e1f4f78821d3deea923a1512a5c340525f9ffdbf5312"} Nov 01 15:48:12.625462 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:12.625442 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" Nov 01 15:48:12.625567 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:12.625483 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerStarted","Data":"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140"} Nov 01 15:48:12.634826 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:12.634585 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-zzpzm" podStartSLOduration=1.423294441 podStartE2EDuration="3.634570539s" totalImagesPullingTime="2.211276098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:12.633820147 +0000 UTC m=+30.336986086" watchObservedRunningTime="2026-11-01 15:48:12.634570539 +0000 UTC m=+30.337736478" Nov 01 15:48:12.663290 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:12.663207 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.6727713250000003 podStartE2EDuration="7.663193271s" totalImagesPullingTime="3.977659274s" totalInitContainerRuntime="12.762672ms" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:12.657889781 +0000 UTC m=+30.361055720" watchObservedRunningTime="2026-11-01 15:48:12.663193271 +0000 UTC m=+30.366359204" Nov 01 15:48:12.676503 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:12.676446 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" podStartSLOduration=2.070986503 podStartE2EDuration="4.676429136s" totalImagesPullingTime="2.605442633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:12.675979756 +0000 UTC m=+30.379145693" watchObservedRunningTime="2026-11-01 15:48:12.676429136 +0000 UTC m=+30.379595072" Nov 01 15:48:15.205712 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:15.205685 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2xm97" Nov 01 15:48:15.639764 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:15.639713 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerStarted","Data":"4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4"} Nov 01 15:48:15.639927 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:15.639818 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerStarted","Data":"65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f"} Nov 01 15:48:15.689465 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:15.689429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:15.759297 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:15.759217 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:16.869631 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:16.869590 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:16.965745 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:16.965706 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:18.638351 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:18.638317 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:18.656998 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:18.656968 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerStarted","Data":"4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67"} Nov 01 15:48:18.657158 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:18.657034 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerStarted","Data":"bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07"} Nov 01 15:48:18.657158 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:18.657063 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerStarted","Data":"890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0"} Nov 01 15:48:18.657158 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:18.657086 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerStarted","Data":"e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6"} Nov 01 15:48:18.734413 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:18.734381 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:18.836378 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:18.836346 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:18.931363 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:18.931268 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:19.018223 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:19.018177 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:19.121109 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:19.121069 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:19.491907 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:19.491869 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:20.142060 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:20.142005 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:20.220200 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:20.220128 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:20.906926 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:20.906893 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:21.750156 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:21.750122 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:21.851558 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:21.851528 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:24.250322 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:24.250217 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:24.268502 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:24.268477 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:24.299180 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:24.299130 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=8.613335896 podStartE2EDuration="14.299116866s" totalImagesPullingTime="5.667662575s" totalInitContainerRuntime="18.118395ms" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:18.701014678 +0000 UTC m=+36.404180652" watchObservedRunningTime="2026-11-01 15:48:24.299116866 +0000 UTC m=+42.002283112" Nov 01 15:48:24.364056 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:24.364022 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:48:24.698414 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:24.698388 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:29.076155 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:29.076126 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:29.076553 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:29.076207 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:29.545692 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:29.545652 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:29.667899 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:29.667861 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:32.524578 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:32.524541 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:32.624939 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:32.624908 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:36.721074 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:36.721028 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:36.790971 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:36.790911 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:37.523386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:37.523354 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:37.590499 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:37.590473 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:44.537869 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:44.537826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:44.638468 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:44.638440 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:45.815677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:45.815645 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:45.916265 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:45.916234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:45.981324 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:45.981300 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:49.081141 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:49.081110 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:49.085032 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:49.085014 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-855bfdd46d-trf7z" Nov 01 15:48:50.723297 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:50.723263 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jv28d-6kkx9"] Nov 01 15:48:50.723725 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:50.723558 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" podUID="a8620858-3d8f-4222-abf0-0b468a12c8d9" containerName="insights-gathering" containerID="cri-o://8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742" gracePeriod=30 Nov 01 15:48:50.960568 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:50.960547 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:48:51.059313 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.059242 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle\") pod \"a8620858-3d8f-4222-abf0-0b468a12c8d9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " Nov 01 15:48:51.059313 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.059298 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path\") pod \"a8620858-3d8f-4222-abf0-0b468a12c8d9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " Nov 01 15:48:51.059597 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.059317 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r\") pod \"a8620858-3d8f-4222-abf0-0b468a12c8d9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " Nov 01 15:48:51.059597 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.059382 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle\") pod \"a8620858-3d8f-4222-abf0-0b468a12c8d9\" (UID: \"a8620858-3d8f-4222-abf0-0b468a12c8d9\") " Nov 01 15:48:51.059719 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.059660 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle" pod "a8620858-3d8f-4222-abf0-0b468a12c8d9" (UID: "a8620858-3d8f-4222-abf0-0b468a12c8d9"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:51.059719 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.059674 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path" pod "a8620858-3d8f-4222-abf0-0b468a12c8d9" (UID: "a8620858-3d8f-4222-abf0-0b468a12c8d9"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:48:51.059830 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.059784 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle" pod "a8620858-3d8f-4222-abf0-0b468a12c8d9" (UID: "a8620858-3d8f-4222-abf0-0b468a12c8d9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:51.061547 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.061525 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r" pod "a8620858-3d8f-4222-abf0-0b468a12c8d9" (UID: "a8620858-3d8f-4222-abf0-0b468a12c8d9"). InnerVolumeSpecName "kube-api-access-pfd6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:51.160542 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.160507 4573 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/a8620858-3d8f-4222-abf0-0b468a12c8d9-archives-path\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:51.160542 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.160534 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfd6r\" (UniqueName: \"kubernetes.io/projected/a8620858-3d8f-4222-abf0-0b468a12c8d9-kube-api-access-pfd6r\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:51.160542 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.160545 4573 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-trusted-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:51.160737 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.160553 4573 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8620858-3d8f-4222-abf0-0b468a12c8d9-service-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:51.356266 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.356191 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:51.722741 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.722709 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:51.774067 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.771877 4573 generic.go:356] "Generic (PLEG): container finished" podID="a8620858-3d8f-4222-abf0-0b468a12c8d9" containerID="8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742" exitCode=2 Nov 01 15:48:51.774067 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.771955 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" event={"ID":"a8620858-3d8f-4222-abf0-0b468a12c8d9","Type":"ContainerDied","Data":"8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742"} Nov 01 15:48:51.774067 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.771998 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" event={"ID":"a8620858-3d8f-4222-abf0-0b468a12c8d9","Type":"ContainerDied","Data":"7e840803a5f170f4f599be0b9523559e3111aaf8d584fe3cad6e0dda3cdc168a"} Nov 01 15:48:51.774067 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.772040 4573 scope.go:122] "RemoveContainer" containerID="8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742" Nov 01 15:48:51.774067 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.772174 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jv28d-6kkx9" Nov 01 15:48:51.783936 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.783917 4573 scope.go:122] "RemoveContainer" containerID="8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742" Nov 01 15:48:51.784202 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:51.784180 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742\": container with ID starting with 8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742 not found: ID does not exist" containerID="8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742" Nov 01 15:48:51.784247 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.784211 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742"} err="failed to get container status \"8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742\": rpc error: code = NotFound desc = could not find container \"8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742\": container with ID starting with 8d12aee511a16e5d60cc576043cc7246b0d255d15ff65da5c7d97b39de5bd742 not found: ID does not exist" Nov 01 15:48:51.793805 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.793782 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jv28d-6kkx9"] Nov 01 15:48:51.797319 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.797300 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-jv28d-6kkx9"] Nov 01 15:48:51.809850 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:51.809830 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:52.331527 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:52.331497 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:52.398659 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:52.398633 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:52.455267 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:52.455228 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8620858-3d8f-4222-abf0-0b468a12c8d9" path="/var/lib/kubelet/pods/a8620858-3d8f-4222-abf0-0b468a12c8d9/volumes" Nov 01 15:48:53.037882 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:53.037848 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:53.141228 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:53.141200 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:48:54.077477 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.077442 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:54.077929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.077710 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="alertmanager" containerID="cri-o://0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8" gracePeriod=120 Nov 01 15:48:54.077929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.077736 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy" containerID="cri-o://56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4" gracePeriod=120 Nov 01 15:48:54.077929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.077771 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy-web" containerID="cri-o://95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282" gracePeriod=120 Nov 01 15:48:54.077929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.077821 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy-metric" containerID="cri-o://c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc" gracePeriod=120 Nov 01 15:48:54.077929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.077806 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="config-reloader" containerID="cri-o://be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413" gracePeriod=120 Nov 01 15:48:54.078164 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.077806 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="prom-label-proxy" containerID="cri-o://d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140" gracePeriod=120 Nov 01 15:48:54.362025 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:54.361991 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:48:54.786654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.786574 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerID="d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140" exitCode=0 Nov 01 15:48:54.786654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.786597 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerID="56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4" exitCode=0 Nov 01 15:48:54.786654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.786606 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerID="be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413" exitCode=0 Nov 01 15:48:54.786654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.786612 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerID="0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8" exitCode=0 Nov 01 15:48:54.786654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.786644 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerDied","Data":"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140"} Nov 01 15:48:54.786948 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.786714 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerDied","Data":"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4"} Nov 01 15:48:54.786948 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.786741 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerDied","Data":"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413"} Nov 01 15:48:54.786948 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:54.786781 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerDied","Data":"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8"} Nov 01 15:48:55.312852 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.312830 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.391030 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.390999 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-kube-api-access-4nvn8\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-kube-api-access-4nvn8\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391197 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391037 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-main-tls\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391197 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391061 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-web\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391197 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391103 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-volume\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-volume\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391370 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391230 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-cluster-tls-config\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391370 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391280 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391370 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391314 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-tls-assets\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-tls-assets\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391370 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391345 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-out\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391580 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391396 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391580 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391462 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-metrics-client-ca\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391580 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391497 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391580 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391525 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-web-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-web-config\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.391580 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.391569 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-main-db\") pod \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\" (UID: \"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce\") " Nov 01 15:48:55.392081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.392057 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-main-db" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:48:55.392621 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.392599 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-metrics-client-ca" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:55.392838 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.392808 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:55.394985 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.394959 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-kube-api-access-4nvn8" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "kube-api-access-4nvn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:55.395077 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.394965 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-main-tls" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:55.395141 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.395096 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:55.395475 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.395452 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-metric" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:55.395599 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.395545 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-web" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:55.395844 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.395804 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-volume" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:55.397099 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.397071 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-tls-assets" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:55.397409 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.397383 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-out" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:48:55.401373 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.401343 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-cluster-tls-config" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:55.408439 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.408393 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-web-config" pod "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" (UID: "e4f0fea5-3aed-40c6-bf06-3215c3ba40ce"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:55.493106 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493076 4573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-main-db\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493106 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493099 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nvn8\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-kube-api-access-4nvn8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493106 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493110 4573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-main-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493122 4573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493133 4573 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493142 4573 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-cluster-tls-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493150 4573 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493158 4573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-tls-assets\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493166 4573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-config-out\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493175 4573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493183 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-metrics-client-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493192 4573 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.493306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.493201 4573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce-web-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:55.792628 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.792541 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerID="c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc" exitCode=0 Nov 01 15:48:55.792628 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.792567 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerID="95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282" exitCode=0 Nov 01 15:48:55.792628 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.792603 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerDied","Data":"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc"} Nov 01 15:48:55.792873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.792632 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.792873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.792658 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerDied","Data":"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282"} Nov 01 15:48:55.792873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.792687 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e4f0fea5-3aed-40c6-bf06-3215c3ba40ce","Type":"ContainerDied","Data":"4b78debb3096b7651249df265b4772d42430761756734ebf1b511724b24cfa09"} Nov 01 15:48:55.792873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.792715 4573 scope.go:122] "RemoveContainer" containerID="d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140" Nov 01 15:48:55.800842 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.800793 4573 scope.go:122] "RemoveContainer" containerID="c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc" Nov 01 15:48:55.807789 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.807770 4573 scope.go:122] "RemoveContainer" containerID="56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4" Nov 01 15:48:55.814659 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.814636 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:55.816360 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.816346 4573 scope.go:122] "RemoveContainer" containerID="95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282" Nov 01 15:48:55.818148 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.818129 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:55.823126 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.823108 4573 scope.go:122] "RemoveContainer" containerID="be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413" Nov 01 15:48:55.829556 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.829538 4573 scope.go:122] "RemoveContainer" containerID="0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8" Nov 01 15:48:55.836134 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.836118 4573 scope.go:122] "RemoveContainer" containerID="5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d" Nov 01 15:48:55.842379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.842364 4573 scope.go:122] "RemoveContainer" containerID="d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140" Nov 01 15:48:55.842618 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:55.842602 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140\": container with ID starting with d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140 not found: ID does not exist" containerID="d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140" Nov 01 15:48:55.842676 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.842625 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140"} err="failed to get container status \"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140\": rpc error: code = NotFound desc = could not find container \"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140\": container with ID starting with d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140 not found: ID does not exist" Nov 01 15:48:55.842676 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.842642 4573 scope.go:122] "RemoveContainer" containerID="c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc" Nov 01 15:48:55.842931 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:55.842912 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc\": container with ID starting with c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc not found: ID does not exist" containerID="c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc" Nov 01 15:48:55.842985 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.842938 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc"} err="failed to get container status \"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc\": rpc error: code = NotFound desc = could not find container \"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc\": container with ID starting with c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc not found: ID does not exist" Nov 01 15:48:55.842985 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.842953 4573 scope.go:122] "RemoveContainer" containerID="56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4" Nov 01 15:48:55.843178 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:55.843164 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4\": container with ID starting with 56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4 not found: ID does not exist" containerID="56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4" Nov 01 15:48:55.843221 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.843180 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4"} err="failed to get container status \"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4\": rpc error: code = NotFound desc = could not find container \"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4\": container with ID starting with 56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4 not found: ID does not exist" Nov 01 15:48:55.843221 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.843191 4573 scope.go:122] "RemoveContainer" containerID="95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282" Nov 01 15:48:55.843423 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:55.843410 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282\": container with ID starting with 95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282 not found: ID does not exist" containerID="95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282" Nov 01 15:48:55.843461 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.843425 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282"} err="failed to get container status \"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282\": rpc error: code = NotFound desc = could not find container \"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282\": container with ID starting with 95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282 not found: ID does not exist" Nov 01 15:48:55.843461 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.843437 4573 scope.go:122] "RemoveContainer" containerID="be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413" Nov 01 15:48:55.843676 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:55.843659 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413\": container with ID starting with be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413 not found: ID does not exist" containerID="be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413" Nov 01 15:48:55.843728 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.843681 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413"} err="failed to get container status \"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413\": rpc error: code = NotFound desc = could not find container \"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413\": container with ID starting with be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413 not found: ID does not exist" Nov 01 15:48:55.843728 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.843701 4573 scope.go:122] "RemoveContainer" containerID="0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8" Nov 01 15:48:55.843933 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:55.843912 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8\": container with ID starting with 0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8 not found: ID does not exist" containerID="0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8" Nov 01 15:48:55.843933 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.843929 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8"} err="failed to get container status \"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8\": rpc error: code = NotFound desc = could not find container \"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8\": container with ID starting with 0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8 not found: ID does not exist" Nov 01 15:48:55.844081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.843940 4573 scope.go:122] "RemoveContainer" containerID="5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d" Nov 01 15:48:55.844119 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:55.844111 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d\": container with ID starting with 5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d not found: ID does not exist" containerID="5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d" Nov 01 15:48:55.844158 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844123 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d"} err="failed to get container status \"5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d\": rpc error: code = NotFound desc = could not find container \"5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d\": container with ID starting with 5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d not found: ID does not exist" Nov 01 15:48:55.844158 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844135 4573 scope.go:122] "RemoveContainer" containerID="d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140" Nov 01 15:48:55.844321 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844303 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140"} err="failed to get container status \"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140\": rpc error: code = NotFound desc = could not find container \"d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140\": container with ID starting with d0b031c55721b2223da7ca0db4c533a42577adb3630c43d97033106472244140 not found: ID does not exist" Nov 01 15:48:55.844321 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844317 4573 scope.go:122] "RemoveContainer" containerID="c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc" Nov 01 15:48:55.844487 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844472 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc"} err="failed to get container status \"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc\": rpc error: code = NotFound desc = could not find container \"c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc\": container with ID starting with c8803a97f9f209a042cc622ab2497756516a5d9ab5e46df047fdef6d5255c8bc not found: ID does not exist" Nov 01 15:48:55.844487 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844485 4573 scope.go:122] "RemoveContainer" containerID="56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4" Nov 01 15:48:55.844689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844673 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4"} err="failed to get container status \"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4\": rpc error: code = NotFound desc = could not find container \"56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4\": container with ID starting with 56caff8d1c78370fc041f4c45ef1c056590c911820ec7e4e77eccb4dc5b1acb4 not found: ID does not exist" Nov 01 15:48:55.844737 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844690 4573 scope.go:122] "RemoveContainer" containerID="95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282" Nov 01 15:48:55.844899 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844883 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282"} err="failed to get container status \"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282\": rpc error: code = NotFound desc = could not find container \"95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282\": container with ID starting with 95b9ce424c7ed1d9836215e105f1f77dce0c990df529804f4e0b1a8576a48282 not found: ID does not exist" Nov 01 15:48:55.844940 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.844899 4573 scope.go:122] "RemoveContainer" containerID="be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413" Nov 01 15:48:55.845081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.845066 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413"} err="failed to get container status \"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413\": rpc error: code = NotFound desc = could not find container \"be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413\": container with ID starting with be9471aa93fad5b070d1736db16ef7b02b22c89d049e46bdac151f00188ba413 not found: ID does not exist" Nov 01 15:48:55.845125 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.845081 4573 scope.go:122] "RemoveContainer" containerID="0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8" Nov 01 15:48:55.845261 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.845246 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8"} err="failed to get container status \"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8\": rpc error: code = NotFound desc = could not find container \"0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8\": container with ID starting with 0f008216d4d182027b5ef5a0e5198163a72f9b44c613a556096f0f642d3048d8 not found: ID does not exist" Nov 01 15:48:55.845303 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.845261 4573 scope.go:122] "RemoveContainer" containerID="5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d" Nov 01 15:48:55.845452 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.845435 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d"} err="failed to get container status \"5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d\": rpc error: code = NotFound desc = could not find container \"5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d\": container with ID starting with 5781703ea91050bc39b77141891ae96c9b1d3b880657de2511b6a5381e51a27d not found: ID does not exist" Nov 01 15:48:55.849888 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.849868 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:55.850115 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850103 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="config-reloader" Nov 01 15:48:55.850161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850116 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="config-reloader" Nov 01 15:48:55.850161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850127 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy" Nov 01 15:48:55.850161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850132 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy" Nov 01 15:48:55.850161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850139 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="init-config-reloader" Nov 01 15:48:55.850161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850144 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="init-config-reloader" Nov 01 15:48:55.850161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850155 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy-web" Nov 01 15:48:55.850161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850160 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy-web" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850165 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a8620858-3d8f-4222-abf0-0b468a12c8d9" containerName="insights-gathering" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850170 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8620858-3d8f-4222-abf0-0b468a12c8d9" containerName="insights-gathering" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850181 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="alertmanager" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850186 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="alertmanager" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850193 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="prom-label-proxy" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850197 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="prom-label-proxy" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850205 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy-metric" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850209 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy-metric" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850247 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="config-reloader" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850254 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850260 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy-web" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850265 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8620858-3d8f-4222-abf0-0b468a12c8d9" containerName="insights-gathering" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850271 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="alertmanager" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850276 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="prom-label-proxy" Nov 01 15:48:55.850386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.850281 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" containerName="kube-rbac-proxy-metric" Nov 01 15:48:55.855365 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.855351 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.866057 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.866028 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:55.879806 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.879786 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 01 15:48:55.893853 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.893833 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 01 15:48:55.893970 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.893910 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q2zk9\"" Nov 01 15:48:55.893970 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.893888 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 01 15:48:55.894081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.893992 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 01 15:48:55.894081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.894020 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 01 15:48:55.894081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.894026 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 01 15:48:55.894081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.894042 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 01 15:48:55.894081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.894056 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 01 15:48:55.894081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.894076 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 01 15:48:55.896305 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896287 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37931145-6e82-4af7-9dcb-32e8bb52146f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896349 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896316 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37931145-6e82-4af7-9dcb-32e8bb52146f-config-out\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896349 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896334 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdl9\" (UniqueName: \"kubernetes.io/projected/37931145-6e82-4af7-9dcb-32e8bb52146f-kube-api-access-dqdl9\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896425 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896411 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37931145-6e82-4af7-9dcb-32e8bb52146f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896458 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896438 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-config-volume\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896487 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896457 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896487 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896474 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896571 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896558 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896607 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896580 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/37931145-6e82-4af7-9dcb-32e8bb52146f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896636 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896607 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-web-config\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896668 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896654 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37931145-6e82-4af7-9dcb-32e8bb52146f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896697 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896683 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.896745 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.896734 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997289 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997263 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-web-config\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997431 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997298 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37931145-6e82-4af7-9dcb-32e8bb52146f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997431 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997315 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997431 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997339 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997431 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997367 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37931145-6e82-4af7-9dcb-32e8bb52146f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997636 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997478 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37931145-6e82-4af7-9dcb-32e8bb52146f-config-out\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997636 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997522 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdl9\" (UniqueName: \"kubernetes.io/projected/37931145-6e82-4af7-9dcb-32e8bb52146f-kube-api-access-dqdl9\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997636 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997572 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37931145-6e82-4af7-9dcb-32e8bb52146f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997636 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997598 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-config-volume\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.997636 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997623 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.998027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997650 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.998027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997689 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.998027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.997717 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/37931145-6e82-4af7-9dcb-32e8bb52146f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.998501 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.998384 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37931145-6e82-4af7-9dcb-32e8bb52146f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.998910 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.998603 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/37931145-6e82-4af7-9dcb-32e8bb52146f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:55.998910 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:55.998864 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/37931145-6e82-4af7-9dcb-32e8bb52146f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.000363 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.000337 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.000458 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.000340 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/37931145-6e82-4af7-9dcb-32e8bb52146f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.000534 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.000466 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/37931145-6e82-4af7-9dcb-32e8bb52146f-config-out\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.000598 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.000549 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.000730 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.000711 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-web-config\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.000883 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.000865 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.001235 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.001216 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-config-volume\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.002098 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.002081 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.002179 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.002147 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/37931145-6e82-4af7-9dcb-32e8bb52146f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.005874 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.005858 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdl9\" (UniqueName: \"kubernetes.io/projected/37931145-6e82-4af7-9dcb-32e8bb52146f-kube-api-access-dqdl9\") pod \"alertmanager-main-0\" (UID: \"37931145-6e82-4af7-9dcb-32e8bb52146f\") " pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.180235 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.180209 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 01 15:48:56.454225 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.454155 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f0fea5-3aed-40c6-bf06-3215c3ba40ce" path="/var/lib/kubelet/pods/e4f0fea5-3aed-40c6-bf06-3215c3ba40ce/volumes" Nov 01 15:48:56.464402 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.464380 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 01 15:48:56.467693 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:48:56.467670 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37931145_6e82_4af7_9dcb_32e8bb52146f.slice/crio-9b2a0a308c79ac0de3ac2442c90831e11710df81f65aff9160c4f69f399feaa5 WatchSource:0}: Error finding container 9b2a0a308c79ac0de3ac2442c90831e11710df81f65aff9160c4f69f399feaa5: Status 404 returned error can't find the container with id 9b2a0a308c79ac0de3ac2442c90831e11710df81f65aff9160c4f69f399feaa5 Nov 01 15:48:56.798840 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.798726 4573 generic.go:356] "Generic (PLEG): container finished" podID="37931145-6e82-4af7-9dcb-32e8bb52146f" containerID="707c33d2cd2661d9bddbdc8106c215ae172021e47c45cf3eb08d4a40f5128e3c" exitCode=0 Nov 01 15:48:56.798840 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.798817 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37931145-6e82-4af7-9dcb-32e8bb52146f","Type":"ContainerDied","Data":"707c33d2cd2661d9bddbdc8106c215ae172021e47c45cf3eb08d4a40f5128e3c"} Nov 01 15:48:56.799021 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:56.798887 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37931145-6e82-4af7-9dcb-32e8bb52146f","Type":"ContainerStarted","Data":"9b2a0a308c79ac0de3ac2442c90831e11710df81f65aff9160c4f69f399feaa5"} Nov 01 15:48:57.806643 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:57.806606 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37931145-6e82-4af7-9dcb-32e8bb52146f","Type":"ContainerStarted","Data":"8ca2d0551e033bf625d30e45e7c0a16ccbd62caf347f5b5049f1968715986958"} Nov 01 15:48:57.807082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:57.806690 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37931145-6e82-4af7-9dcb-32e8bb52146f","Type":"ContainerStarted","Data":"64bfd215c1d707eecb7a453c4042c6f42a4d72bb6e04c9925791324129fd8b90"} Nov 01 15:48:57.807082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:57.806716 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37931145-6e82-4af7-9dcb-32e8bb52146f","Type":"ContainerStarted","Data":"90a344e476f69e61d8e7a0391426472bd17a7f910da2adc33b10fa59631e2024"} Nov 01 15:48:57.807082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:57.806742 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37931145-6e82-4af7-9dcb-32e8bb52146f","Type":"ContainerStarted","Data":"e9d3884bd59dafbd9e2eea627f1e31808c93b18a76627655832af87cb7508caa"} Nov 01 15:48:57.807082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:57.806785 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37931145-6e82-4af7-9dcb-32e8bb52146f","Type":"ContainerStarted","Data":"8a0474b5dac474e2140aa2aa054b7af08cb7ec3fbe2bea3c5c0b111d943204e8"} Nov 01 15:48:57.807082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:57.806808 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"37931145-6e82-4af7-9dcb-32e8bb52146f","Type":"ContainerStarted","Data":"f5c755bd39412724e2add52ca43a1b7cb32f38645e81f065befbcba0518a89c3"} Nov 01 15:48:57.833141 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:57.832459 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.8196981230000002 podStartE2EDuration="2.832443555s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.745432ms" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:48:57.830949831 +0000 UTC m=+75.534115773" watchObservedRunningTime="2026-11-01 15:48:57.832443555 +0000 UTC m=+75.535609495" Nov 01 15:48:58.092219 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.092127 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6755758fcb-2nfbk"] Nov 01 15:48:58.095395 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.095379 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.106351 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.106327 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Nov 01 15:48:58.106532 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.106519 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Nov 01 15:48:58.106584 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.106529 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Nov 01 15:48:58.106640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.106606 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Nov 01 15:48:58.106640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.106636 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Nov 01 15:48:58.107671 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.107643 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-b9j7r\"" Nov 01 15:48:58.109370 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.109349 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Nov 01 15:48:58.110471 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.110451 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6755758fcb-2nfbk"] Nov 01 15:48:58.213070 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.213039 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-secret-telemeter-client\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.213070 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.213070 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.213252 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.213105 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-metrics-client-ca\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.213252 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.213205 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhdj\" (UniqueName: \"kubernetes.io/projected/8174b90f-0f97-4b31-9756-faa476cfcbb8-kube-api-access-2qhdj\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.213252 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.213239 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-serving-certs-ca-bundle\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.213381 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.213254 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-telemeter-client-tls\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.213381 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.213270 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.213381 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.213365 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-federate-client-tls\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.314338 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.314304 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-metrics-client-ca\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.314469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.314354 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhdj\" (UniqueName: \"kubernetes.io/projected/8174b90f-0f97-4b31-9756-faa476cfcbb8-kube-api-access-2qhdj\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.314469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.314376 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-serving-certs-ca-bundle\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.314469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.314393 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-telemeter-client-tls\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.314469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.314410 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.314469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.314445 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-federate-client-tls\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.314469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.314465 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-secret-telemeter-client\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.314809 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.314481 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.315234 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.315208 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-serving-certs-ca-bundle\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.315331 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.315208 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-metrics-client-ca\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.315391 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.315333 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8174b90f-0f97-4b31-9756-faa476cfcbb8-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.317075 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.317053 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-telemeter-client-tls\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.317186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.317102 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-secret-telemeter-client\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.317322 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.317305 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.317396 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.317380 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8174b90f-0f97-4b31-9756-faa476cfcbb8-federate-client-tls\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.321761 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.321729 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhdj\" (UniqueName: \"kubernetes.io/projected/8174b90f-0f97-4b31-9756-faa476cfcbb8-kube-api-access-2qhdj\") pod \"telemeter-client-6755758fcb-2nfbk\" (UID: \"8174b90f-0f97-4b31-9756-faa476cfcbb8\") " pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.411930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.411910 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" Nov 01 15:48:58.444059 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.444027 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:48:58.444457 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.444405 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="prometheus" containerID="cri-o://65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f" gracePeriod=600 Nov 01 15:48:58.444540 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.444469 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy-web" containerID="cri-o://890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0" gracePeriod=600 Nov 01 15:48:58.444540 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.444501 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="thanos-sidecar" containerID="cri-o://e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6" gracePeriod=600 Nov 01 15:48:58.444540 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.444517 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="config-reloader" containerID="cri-o://4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4" gracePeriod=600 Nov 01 15:48:58.444693 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.444480 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy" containerID="cri-o://bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07" gracePeriod=600 Nov 01 15:48:58.444693 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.444472 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy-thanos" containerID="cri-o://4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67" gracePeriod=600 Nov 01 15:48:58.693840 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.693775 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6755758fcb-2nfbk"] Nov 01 15:48:58.696619 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:48:58.696595 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8174b90f_0f97_4b31_9756_faa476cfcbb8.slice/crio-c63e9941f1cc81fe06dfb0e238f184d97e07246ef51548d8c3599df62c2e27e8 WatchSource:0}: Error finding container c63e9941f1cc81fe06dfb0e238f184d97e07246ef51548d8c3599df62c2e27e8: Status 404 returned error can't find the container with id c63e9941f1cc81fe06dfb0e238f184d97e07246ef51548d8c3599df62c2e27e8 Nov 01 15:48:58.813257 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813223 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerID="4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67" exitCode=0 Nov 01 15:48:58.813257 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813247 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerID="bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07" exitCode=0 Nov 01 15:48:58.813257 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813256 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerID="e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6" exitCode=0 Nov 01 15:48:58.813257 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813264 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerID="4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4" exitCode=0 Nov 01 15:48:58.813786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813271 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerID="65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f" exitCode=0 Nov 01 15:48:58.813786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813294 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerDied","Data":"4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67"} Nov 01 15:48:58.813786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813377 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerDied","Data":"bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07"} Nov 01 15:48:58.813786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813423 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerDied","Data":"e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6"} Nov 01 15:48:58.813786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813462 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerDied","Data":"4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4"} Nov 01 15:48:58.813786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.813501 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerDied","Data":"65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f"} Nov 01 15:48:58.814500 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:58.814478 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" event={"ID":"8174b90f-0f97-4b31-9756-faa476cfcbb8","Type":"ContainerStarted","Data":"c63e9941f1cc81fe06dfb0e238f184d97e07246ef51548d8c3599df62c2e27e8"} Nov 01 15:48:59.684038 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.683993 4573 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 01 15:48:59.684162 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.684066 4573 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 01 15:48:59.684162 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.684095 4573 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Nov 01 15:48:59.684162 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.684120 4573 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Nov 01 15:48:59.713211 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.713177 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:59.820551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.820521 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerID="890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0" exitCode=0 Nov 01 15:48:59.820911 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.820581 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerDied","Data":"890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0"} Nov 01 15:48:59.820911 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.820600 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:48:59.820911 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.820637 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff186e3e-3101-466a-b675-a6cfe3d61410","Type":"ContainerDied","Data":"70a401034168bfd0f962d5bed8d380cfc4c7285934f4719027140b0ec685309a"} Nov 01 15:48:59.820911 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.820669 4573 scope.go:122] "RemoveContainer" containerID="4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67" Nov 01 15:48:59.828824 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.828798 4573 scope.go:122] "RemoveContainer" containerID="bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07" Nov 01 15:48:59.829009 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.828982 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-serving-certs-ca-bundle\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829127 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829046 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829188 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829136 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-db\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829188 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829179 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-trusted-ca-bundle\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829241 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-config\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829284 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-kubelet-serving-ca-bundle\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829311 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-tls-assets\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-tls-assets\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829343 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-metrics-client-certs\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829363 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-serving-certs-ca-bundle" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829390 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-1\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829418 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-tls\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829452 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-kube-api-access-29jt5\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-kube-api-access-29jt5\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829491 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-kube-rbac-proxy\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829516 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-thanos-prometheus-http-client-file\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.829551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829549 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.830196 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.829781 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-kubelet-serving-ca-bundle" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:59.830196 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830029 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-trusted-ca-bundle" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:59.830316 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830297 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-config-out\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.830374 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830339 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-2\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.830430 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830387 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-metrics-client-ca\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.830430 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830398 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-1" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:59.830430 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830415 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-grpc-tls\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.830559 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830449 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-0\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.830559 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830478 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-web-config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-web-config\") pod \"ff186e3e-3101-466a-b675-a6cfe3d61410\" (UID: \"ff186e3e-3101-466a-b675-a6cfe3d61410\") " Nov 01 15:48:59.830721 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830708 4573 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.830802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830730 4573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.830802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830761 4573 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.830802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.830778 4573 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.831903 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.831877 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-db" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:48:59.832630 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.832601 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-metrics-client-ca" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:59.832708 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.832666 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-2" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:59.833700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.833674 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-0" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:48:59.834587 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.834550 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-tls-assets" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:59.834955 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.834914 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-config-out" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:48:59.835216 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.835184 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-kube-api-access-29jt5" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "kube-api-access-29jt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:48:59.836264 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.836171 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-thanos-sidecar-tls" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.836373 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.836351 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-config" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.836460 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.836438 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-grpc-tls" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.836553 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.836523 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-kube-rbac-proxy" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.836639 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.836610 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-kube-rbac-proxy-web" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.836847 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.836825 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-thanos-prometheus-http-client-file" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.836981 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.836947 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-tls" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.837181 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.837163 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-metrics-client-certs" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.846448 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.846422 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-web-config" pod "ff186e3e-3101-466a-b675-a6cfe3d61410" (UID: "ff186e3e-3101-466a-b675-a6cfe3d61410"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:48:59.850824 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.850803 4573 scope.go:122] "RemoveContainer" containerID="890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0" Nov 01 15:48:59.858084 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.858068 4573 scope.go:122] "RemoveContainer" containerID="e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6" Nov 01 15:48:59.865059 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.865039 4573 scope.go:122] "RemoveContainer" containerID="4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4" Nov 01 15:48:59.872481 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.872457 4573 scope.go:122] "RemoveContainer" containerID="65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f" Nov 01 15:48:59.880696 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.880678 4573 scope.go:122] "RemoveContainer" containerID="a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1" Nov 01 15:48:59.888268 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.888253 4573 scope.go:122] "RemoveContainer" containerID="4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67" Nov 01 15:48:59.888570 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.888541 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67\": container with ID starting with 4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67 not found: ID does not exist" containerID="4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67" Nov 01 15:48:59.888654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.888583 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67"} err="failed to get container status \"4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67\": rpc error: code = NotFound desc = could not find container \"4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67\": container with ID starting with 4651a0672de2f06ee0da39160976f6e0a16caaca63d847e2fdbcecfd7277fa67 not found: ID does not exist" Nov 01 15:48:59.888654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.888601 4573 scope.go:122] "RemoveContainer" containerID="bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07" Nov 01 15:48:59.888924 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.888904 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07\": container with ID starting with bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07 not found: ID does not exist" containerID="bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07" Nov 01 15:48:59.889002 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.888935 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07"} err="failed to get container status \"bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07\": rpc error: code = NotFound desc = could not find container \"bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07\": container with ID starting with bb0fa331f7a6edb8a52dbcf88d626cb7eb8ac107b498cdf8f6720f60f1c69a07 not found: ID does not exist" Nov 01 15:48:59.889002 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.888960 4573 scope.go:122] "RemoveContainer" containerID="890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0" Nov 01 15:48:59.889243 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.889226 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0\": container with ID starting with 890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0 not found: ID does not exist" containerID="890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0" Nov 01 15:48:59.889310 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.889249 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0"} err="failed to get container status \"890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0\": rpc error: code = NotFound desc = could not find container \"890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0\": container with ID starting with 890a301057f585bab46e692e10fd2c49dfca8ae839dcb0cb7d58196bed96edb0 not found: ID does not exist" Nov 01 15:48:59.889310 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.889268 4573 scope.go:122] "RemoveContainer" containerID="e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6" Nov 01 15:48:59.889525 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.889503 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6\": container with ID starting with e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6 not found: ID does not exist" containerID="e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6" Nov 01 15:48:59.889615 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.889531 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6"} err="failed to get container status \"e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6\": rpc error: code = NotFound desc = could not find container \"e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6\": container with ID starting with e03d6f884482674afb1aa06cc633096e588ae6815170681204e9c4aa2d05f8d6 not found: ID does not exist" Nov 01 15:48:59.889615 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.889551 4573 scope.go:122] "RemoveContainer" containerID="4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4" Nov 01 15:48:59.889828 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.889804 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4\": container with ID starting with 4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4 not found: ID does not exist" containerID="4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4" Nov 01 15:48:59.889882 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.889835 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4"} err="failed to get container status \"4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4\": rpc error: code = NotFound desc = could not find container \"4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4\": container with ID starting with 4cbd4c407b7050750d6476384fc66242d83afc0f0a69d4f71fc245a6b855c8f4 not found: ID does not exist" Nov 01 15:48:59.889882 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.889850 4573 scope.go:122] "RemoveContainer" containerID="65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f" Nov 01 15:48:59.890129 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.890104 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f\": container with ID starting with 65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f not found: ID does not exist" containerID="65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f" Nov 01 15:48:59.890186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.890138 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f"} err="failed to get container status \"65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f\": rpc error: code = NotFound desc = could not find container \"65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f\": container with ID starting with 65c594e7b13717224da40c53f0ac55ffad2443cc296589a925948c7a64ae5f2f not found: ID does not exist" Nov 01 15:48:59.890186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.890158 4573 scope.go:122] "RemoveContainer" containerID="a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1" Nov 01 15:48:59.890462 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:48:59.890440 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1\": container with ID starting with a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1 not found: ID does not exist" containerID="a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1" Nov 01 15:48:59.890546 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.890470 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1"} err="failed to get container status \"a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1\": rpc error: code = NotFound desc = could not find container \"a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1\": container with ID starting with a9caf292f16919d26efa17cbe470ffac5533e3abcfee7e87bb255685422472a1 not found: ID does not exist" Nov 01 15:48:59.932212 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932175 4573 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-configmap-metrics-client-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932212 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932211 4573 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-grpc-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932227 4573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932242 4573 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-web-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932258 4573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932273 4573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-db\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932287 4573 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-config\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932300 4573 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-tls-assets\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932314 4573 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-metrics-client-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932329 4573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932344 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29jt5\" (UniqueName: \"kubernetes.io/projected/ff186e3e-3101-466a-b675-a6cfe3d61410-kube-api-access-29jt5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932358 4573 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-kube-rbac-proxy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932711 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932374 4573 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932711 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932390 4573 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff186e3e-3101-466a-b675-a6cfe3d61410-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932711 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932404 4573 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff186e3e-3101-466a-b675-a6cfe3d61410-config-out\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:48:59.932711 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:48:59.932417 4573 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff186e3e-3101-466a-b675-a6cfe3d61410-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:49:00.148359 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.148319 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:49:00.152543 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.152517 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:49:00.176191 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176166 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:49:00.176478 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176453 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="config-reloader" Nov 01 15:49:00.176478 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176474 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="config-reloader" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176488 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="thanos-sidecar" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176494 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="thanos-sidecar" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176500 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy-web" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176505 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy-web" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176523 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="prometheus" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176528 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="prometheus" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176534 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176539 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176546 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="init-config-reloader" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176553 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="init-config-reloader" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176562 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy-thanos" Nov 01 15:49:00.176589 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176570 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy-thanos" Nov 01 15:49:00.177027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176621 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="config-reloader" Nov 01 15:49:00.177027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176632 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="thanos-sidecar" Nov 01 15:49:00.177027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176641 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy-web" Nov 01 15:49:00.177027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176652 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="prometheus" Nov 01 15:49:00.177027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176661 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy" Nov 01 15:49:00.177027 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.176670 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" containerName="kube-rbac-proxy-thanos" Nov 01 15:49:00.182099 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.182078 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.185666 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.185644 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 15:49:00.185808 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.185767 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 15:49:00.185852 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.185809 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 15:49:00.185979 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.185950 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 15:49:00.186085 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.185986 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 15:49:00.186085 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.185995 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 15:49:00.186199 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.186184 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 15:49:00.186255 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.186187 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 15:49:00.186255 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.186207 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 15:49:00.186338 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.186304 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 15:49:00.186556 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.186536 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 15:49:00.186688 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.186669 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-hx8tz\"" Nov 01 15:49:00.187031 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.187007 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 15:49:00.187031 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.187024 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ausg30f60sle\"" Nov 01 15:49:00.188946 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.188925 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 15:49:00.189119 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.189099 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 15:49:00.200500 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.200419 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:49:00.235135 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235105 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d402a470-6adc-4cd1-b435-d2934102977d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235290 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235152 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235290 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235175 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-web-config\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235398 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235308 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235398 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235371 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235499 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235430 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235499 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235456 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235499 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235484 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-config\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235599 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235559 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235602 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235621 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235690 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235637 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d402a470-6adc-4cd1-b435-d2934102977d-config-out\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235690 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235655 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235690 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235672 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235848 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235720 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235848 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235821 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235955 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235858 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235955 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235881 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235955 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235913 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.235955 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.235950 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhwjt\" (UniqueName: \"kubernetes.io/projected/d402a470-6adc-4cd1-b435-d2934102977d-kube-api-access-nhwjt\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.336724 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.336686 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.336838 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.336733 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.336886 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.336856 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.336929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.336887 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.336929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.336915 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-config\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337019 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.336943 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337019 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.336991 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337113 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337023 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337113 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337048 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d402a470-6adc-4cd1-b435-d2934102977d-config-out\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337113 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337077 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337113 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337103 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337133 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337160 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337201 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337225 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337296 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337260 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337308 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhwjt\" (UniqueName: \"kubernetes.io/projected/d402a470-6adc-4cd1-b435-d2934102977d-kube-api-access-nhwjt\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337314 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337344 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d402a470-6adc-4cd1-b435-d2934102977d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337376 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337422 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-web-config\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337460 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337847 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337639 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.337905 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.337888 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.339449 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.339415 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-config\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.339740 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.339718 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.340640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.340158 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.340640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.340247 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d402a470-6adc-4cd1-b435-d2934102977d-config-out\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.340640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.340308 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.340640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.340392 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-web-config\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.340640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.340475 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.340640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.340596 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.341035 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.340997 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.341350 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.341109 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d402a470-6adc-4cd1-b435-d2934102977d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.342692 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.342663 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.342868 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.342849 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.343024 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.343004 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.343779 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.343741 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d402a470-6adc-4cd1-b435-d2934102977d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.343897 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.343878 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d402a470-6adc-4cd1-b435-d2934102977d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.348705 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.348684 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhwjt\" (UniqueName: \"kubernetes.io/projected/d402a470-6adc-4cd1-b435-d2934102977d-kube-api-access-nhwjt\") pod \"prometheus-k8s-0\" (UID: \"d402a470-6adc-4cd1-b435-d2934102977d\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.461741 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.461379 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff186e3e-3101-466a-b675-a6cfe3d61410" path="/var/lib/kubelet/pods/ff186e3e-3101-466a-b675-a6cfe3d61410/volumes" Nov 01 15:49:00.509135 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.509109 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:00.791366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.791283 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 15:49:00.794017 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:49:00.793993 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd402a470_6adc_4cd1_b435_d2934102977d.slice/crio-d4e6347e775e22c89b71042c665321df78da7a289acf4ccb04140089b01fcf35 WatchSource:0}: Error finding container d4e6347e775e22c89b71042c665321df78da7a289acf4ccb04140089b01fcf35: Status 404 returned error can't find the container with id d4e6347e775e22c89b71042c665321df78da7a289acf4ccb04140089b01fcf35 Nov 01 15:49:00.827531 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.827505 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" event={"ID":"8174b90f-0f97-4b31-9756-faa476cfcbb8","Type":"ContainerStarted","Data":"3b4d2656d2ec2ae976c8d0de0bad826a0ff1b91190dfed7cd7821a8801f8d8d3"} Nov 01 15:49:00.827901 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.827557 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" event={"ID":"8174b90f-0f97-4b31-9756-faa476cfcbb8","Type":"ContainerStarted","Data":"8bf2f9970bfb50705b75cea82535938dae71315259f5c683a5e45b43c3ce8634"} Nov 01 15:49:00.827901 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.827587 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" event={"ID":"8174b90f-0f97-4b31-9756-faa476cfcbb8","Type":"ContainerStarted","Data":"e551746166134953a17a8c4435a1ba3300cd8298453b66024ce7f40076ba18c1"} Nov 01 15:49:00.828723 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.828702 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerStarted","Data":"d4e6347e775e22c89b71042c665321df78da7a289acf4ccb04140089b01fcf35"} Nov 01 15:49:00.848567 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.848516 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6755758fcb-2nfbk" podStartSLOduration=1.244468747 podStartE2EDuration="2.848498956s" totalImagesPullingTime="1.604030209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:00.846918889 +0000 UTC m=+78.550084824" watchObservedRunningTime="2026-11-01 15:49:00.848498956 +0000 UTC m=+78.551664893" Nov 01 15:49:00.994718 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:00.994682 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:01.060739 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:01.060671 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:01.581360 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:01.581325 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:01.681256 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:01.681221 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:01.833342 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:01.833258 4573 generic.go:356] "Generic (PLEG): container finished" podID="d402a470-6adc-4cd1-b435-d2934102977d" containerID="267c49493a09db8935be029d19f0a2f4168e596bb3169d0dee943bb98551bee4" exitCode=0 Nov 01 15:49:01.833825 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:01.833342 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerDied","Data":"267c49493a09db8935be029d19f0a2f4168e596bb3169d0dee943bb98551bee4"} Nov 01 15:49:02.839529 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.839502 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 15:49:02.839988 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.839889 4573 generic.go:356] "Generic (PLEG): container finished" podID="d402a470-6adc-4cd1-b435-d2934102977d" containerID="47a5a22ee4c011e1d01a166b642c382f09f8e956c0e528ef81eb7a9b02086800" exitCode=2 Nov 01 15:49:02.839988 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.839976 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerStarted","Data":"7658420c446adb6e2701c73ad74434d042c13b25685a1190a5eda2311e4e3cc0"} Nov 01 15:49:02.840118 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.840064 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerStarted","Data":"8529228a8229f9e23a393829209848a9f100aa67d1afb9611242d3c78af1932a"} Nov 01 15:49:02.840118 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.840093 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerStarted","Data":"8ebe5d64c3ebfb73be7541b4e6f847d5d348fbde75e5cd86eaf2b1e06b882c0c"} Nov 01 15:49:02.840207 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.840119 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerStarted","Data":"edfecd7c163a1fe41edd26b543cabaa8346bef0f431792cf0dcfd86f614ed9f0"} Nov 01 15:49:02.840207 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.840143 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerStarted","Data":"a568173aadadff816c9477c21221409f78bfa882c90a7dc5c04c9ba4f4cd44c6"} Nov 01 15:49:02.840207 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.840169 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerDied","Data":"47a5a22ee4c011e1d01a166b642c382f09f8e956c0e528ef81eb7a9b02086800"} Nov 01 15:49:02.840403 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:02.840391 4573 scope.go:122] "RemoveContainer" containerID="47a5a22ee4c011e1d01a166b642c382f09f8e956c0e528ef81eb7a9b02086800" Nov 01 15:49:03.846312 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:03.846288 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 15:49:03.846695 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:03.846640 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d402a470-6adc-4cd1-b435-d2934102977d","Type":"ContainerStarted","Data":"576d5aaef239feb78652f08cf8c6c3cfe1944ee307d10e16996c022b01312b9d"} Nov 01 15:49:03.882197 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:03.882143 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.865179736 podStartE2EDuration="3.882130469s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.950733ms" isStatefulPod=false podCreationTimestamp="2026-11-01 15:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:49:03.880057194 +0000 UTC m=+81.583223163" watchObservedRunningTime="2026-11-01 15:49:03.882130469 +0000 UTC m=+81.585296405" Nov 01 15:49:05.509649 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:05.509610 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:49:24.362279 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:49:24.362241 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:49:29.582355 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:29.582319 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:49:54.361868 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:49:54.361786 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:49:57.417087 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:49:57.417050 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:00.509281 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:00.509236 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:50:00.524428 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:00.524397 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:50:01.060826 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:01.060795 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 15:50:08.364836 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:08.364799 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:08.432395 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:08.432354 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:18.065225 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:18.065187 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:18.143643 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:18.143612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:18.806332 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:18.806299 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:20.436162 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:20.436125 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:20.538950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:20.538918 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:20.996729 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:20.996699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:21.069822 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:21.069787 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:24.364855 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:50:24.364814 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:50:29.638385 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:29.638350 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:36.731244 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:36.731202 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:36.830151 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:36.830119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:36.896906 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:36.896877 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:50.834956 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:50:50.834919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:50:54.387709 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:50:54.387671 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:51:24.362515 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:51:24.362430 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:51:29.686416 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:51:29.686382 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:39.458956 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:51:39.458921 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:51:54.362058 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:51:54.362009 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:52:24.361885 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:52:24.361849 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:52:25.051646 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:25.051606 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:26.380794 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:26.380734 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:26.674906 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:26.674831 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:26.767264 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:26.767237 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:27.707952 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:27.707921 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:29.742054 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:29.742021 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:30.649548 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:30.649516 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:30.789814 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:30.789779 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:30.940175 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:30.940107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:31.030153 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:31.030121 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:32.039353 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.039323 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc"] Nov 01 15:52:32.042890 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.042874 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.047349 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.047328 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 15:52:32.047440 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.047402 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-x6pqc\"" Nov 01 15:52:32.049519 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.049497 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 15:52:32.050857 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.050836 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc"] Nov 01 15:52:32.161473 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.161437 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffj8p\" (UniqueName: \"kubernetes.io/projected/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-kube-api-access-ffj8p\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.161637 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.161473 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.161637 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.161564 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.262279 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.262234 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffj8p\" (UniqueName: \"kubernetes.io/projected/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-kube-api-access-ffj8p\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.262279 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.262282 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.262533 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.262330 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.262694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.262671 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.262792 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.262719 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.270460 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.270436 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffj8p\" (UniqueName: \"kubernetes.io/projected/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-kube-api-access-ffj8p\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.335625 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.335552 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2"] Nov 01 15:52:32.338991 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.338971 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.359829 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.359802 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:32.362141 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.362106 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-trnjw\"" Nov 01 15:52:32.363287 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.363267 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2"] Nov 01 15:52:32.464086 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.464052 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.464242 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.464114 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djgbj\" (UniqueName: \"kubernetes.io/projected/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-kube-api-access-djgbj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.464242 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.464185 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.565479 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.565450 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djgbj\" (UniqueName: \"kubernetes.io/projected/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-kube-api-access-djgbj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.565641 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.565482 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.565641 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.565539 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.565936 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.565916 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.565998 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.565980 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.573675 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.573653 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djgbj\" (UniqueName: \"kubernetes.io/projected/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-kube-api-access-djgbj\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.630851 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.630825 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc"] Nov 01 15:52:32.634057 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:52:32.634033 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0400f5_aa65_4d7a_9f83_6235cbfa2b57.slice/crio-350fafd606e4189ced032aad685426c877b437ff80ac0c8d089c9799d4792c8e WatchSource:0}: Error finding container 350fafd606e4189ced032aad685426c877b437ff80ac0c8d089c9799d4792c8e: Status 404 returned error can't find the container with id 350fafd606e4189ced032aad685426c877b437ff80ac0c8d089c9799d4792c8e Nov 01 15:52:32.655061 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.655040 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:32.926645 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:32.926570 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2"] Nov 01 15:52:32.929767 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:52:32.929735 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda142d1ff_7c66_48a1_b7a1_94f661f3d8ae.slice/crio-8eba96f6e1c26cdae705f24494df9e0d35a361b7b1acfdb6d5c356004cbecb8e WatchSource:0}: Error finding container 8eba96f6e1c26cdae705f24494df9e0d35a361b7b1acfdb6d5c356004cbecb8e: Status 404 returned error can't find the container with id 8eba96f6e1c26cdae705f24494df9e0d35a361b7b1acfdb6d5c356004cbecb8e Nov 01 15:52:33.553261 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:33.553219 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:33.556184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:33.556159 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" event={"ID":"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57","Type":"ContainerStarted","Data":"350fafd606e4189ced032aad685426c877b437ff80ac0c8d089c9799d4792c8e"} Nov 01 15:52:33.557631 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:33.557604 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" event={"ID":"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae","Type":"ContainerStarted","Data":"8eba96f6e1c26cdae705f24494df9e0d35a361b7b1acfdb6d5c356004cbecb8e"} Nov 01 15:52:39.584300 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:39.584267 4573 generic.go:356] "Generic (PLEG): container finished" podID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerID="343baeaed3062b00a322cf5c162183f69bd6022c7804f7054917cc15f02dc07a" exitCode=0 Nov 01 15:52:39.584736 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:39.584358 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" event={"ID":"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57","Type":"ContainerDied","Data":"343baeaed3062b00a322cf5c162183f69bd6022c7804f7054917cc15f02dc07a"} Nov 01 15:52:39.585997 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:39.585973 4573 generic.go:356] "Generic (PLEG): container finished" podID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerID="b4c961400911889a6fa44e0eb327da26521856920c80d374bf6001a2a71ed3d8" exitCode=0 Nov 01 15:52:39.586077 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:39.586048 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" event={"ID":"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae","Type":"ContainerDied","Data":"b4c961400911889a6fa44e0eb327da26521856920c80d374bf6001a2a71ed3d8"} Nov 01 15:52:40.592536 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:40.592498 4573 generic.go:356] "Generic (PLEG): container finished" podID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerID="ccc2a118988a807a200e9ce8a2333075d6c7ed0dff113f3fa22f3e5bf5eefda4" exitCode=0 Nov 01 15:52:40.592983 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:40.592552 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" event={"ID":"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae","Type":"ContainerDied","Data":"ccc2a118988a807a200e9ce8a2333075d6c7ed0dff113f3fa22f3e5bf5eefda4"} Nov 01 15:52:42.463103 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:42.463071 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 15:52:42.465469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:42.465441 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 15:52:42.477024 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:42.477000 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:52:42.479839 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:42.479822 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:52:42.601810 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:42.601770 4573 generic.go:356] "Generic (PLEG): container finished" podID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerID="fcd882f3476233b815fd6c57884bea9aa7ec5364726d6a87b54bc8649a597291" exitCode=0 Nov 01 15:52:42.602001 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:42.601849 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" event={"ID":"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57","Type":"ContainerDied","Data":"fcd882f3476233b815fd6c57884bea9aa7ec5364726d6a87b54bc8649a597291"} Nov 01 15:52:47.625813 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:47.625775 4573 generic.go:356] "Generic (PLEG): container finished" podID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerID="7fd5a5b9d450e00511db7d4bf7c9ae94d4daa0aae09e250a00c0989989a59d93" exitCode=0 Nov 01 15:52:47.626253 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:47.625860 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" event={"ID":"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae","Type":"ContainerDied","Data":"7fd5a5b9d450e00511db7d4bf7c9ae94d4daa0aae09e250a00c0989989a59d93"} Nov 01 15:52:48.766474 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:48.766449 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:48.918893 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:48.918866 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-util\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-util\") pod \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " Nov 01 15:52:48.919000 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:48.918924 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-kube-api-access-djgbj\" (UniqueName: \"kubernetes.io/projected/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-kube-api-access-djgbj\") pod \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " Nov 01 15:52:48.919062 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:48.918998 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-bundle\") pod \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\" (UID: \"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae\") " Nov 01 15:52:48.919457 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:48.919434 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-bundle" pod "a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" (UID: "a142d1ff-7c66-48a1-b7a1-94f661f3d8ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:48.920957 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:48.920929 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-kube-api-access-djgbj" pod "a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" (UID: "a142d1ff-7c66-48a1-b7a1-94f661f3d8ae"). InnerVolumeSpecName "kube-api-access-djgbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:48.923989 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:48.923966 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-util" pod "a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" (UID: "a142d1ff-7c66-48a1-b7a1-94f661f3d8ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:49.020153 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:49.020123 4573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-util\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:49.020153 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:49.020152 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djgbj\" (UniqueName: \"kubernetes.io/projected/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-kube-api-access-djgbj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:49.020292 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:49.020167 4573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a142d1ff-7c66-48a1-b7a1-94f661f3d8ae-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:49.634942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:49.634912 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" Nov 01 15:52:49.634942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:49.634940 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqgh2" event={"ID":"a142d1ff-7c66-48a1-b7a1-94f661f3d8ae","Type":"ContainerDied","Data":"8eba96f6e1c26cdae705f24494df9e0d35a361b7b1acfdb6d5c356004cbecb8e"} Nov 01 15:52:49.635191 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:49.634982 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8eba96f6e1c26cdae705f24494df9e0d35a361b7b1acfdb6d5c356004cbecb8e" Nov 01 15:52:49.637051 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:49.637021 4573 generic.go:356] "Generic (PLEG): container finished" podID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerID="ca4eb3be4c191272753ea43a0d40737cd09d4a23443b026c03f40f4c66d7d98e" exitCode=0 Nov 01 15:52:49.637162 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:49.637075 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" event={"ID":"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57","Type":"ContainerDied","Data":"ca4eb3be4c191272753ea43a0d40737cd09d4a23443b026c03f40f4c66d7d98e"} Nov 01 15:52:50.763870 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:50.763842 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:50.936452 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:50.936377 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-bundle\") pod \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " Nov 01 15:52:50.936452 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:50.936434 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-kube-api-access-ffj8p\" (UniqueName: \"kubernetes.io/projected/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-kube-api-access-ffj8p\") pod \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " Nov 01 15:52:50.936632 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:50.936505 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-util\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-util\") pod \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\" (UID: \"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57\") " Nov 01 15:52:50.936782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:50.936739 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-bundle" pod "7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" (UID: "7f0400f5-aa65-4d7a-9f83-6235cbfa2b57"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:50.938620 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:50.938591 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-kube-api-access-ffj8p" pod "7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" (UID: "7f0400f5-aa65-4d7a-9f83-6235cbfa2b57"). InnerVolumeSpecName "kube-api-access-ffj8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:52:50.940684 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:50.940660 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-util" pod "7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" (UID: "7f0400f5-aa65-4d7a-9f83-6235cbfa2b57"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 15:52:51.037498 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:51.037473 4573 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-util\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:51.037498 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:51.037499 4573 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-bundle\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:51.037670 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:51.037514 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffj8p\" (UniqueName: \"kubernetes.io/projected/7f0400f5-aa65-4d7a-9f83-6235cbfa2b57-kube-api-access-ffj8p\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:52:51.645592 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:51.645563 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" Nov 01 15:52:51.645734 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:51.645565 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644fvwpc" event={"ID":"7f0400f5-aa65-4d7a-9f83-6235cbfa2b57","Type":"ContainerDied","Data":"350fafd606e4189ced032aad685426c877b437ff80ac0c8d089c9799d4792c8e"} Nov 01 15:52:51.645734 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:51.645678 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="350fafd606e4189ced032aad685426c877b437ff80ac0c8d089c9799d4792c8e" Nov 01 15:52:52.220196 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:52.220165 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:52.314709 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:52.314678 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:53.981446 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981370 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx"] Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981648 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerName="pull" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981658 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerName="pull" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981667 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerName="extract" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981672 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerName="extract" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981680 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerName="pull" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981685 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerName="pull" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981690 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerName="extract" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981694 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerName="extract" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981703 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerName="util" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981707 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerName="util" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981719 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerName="util" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981724 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerName="util" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981815 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a142d1ff-7c66-48a1-b7a1-94f661f3d8ae" containerName="extract" Nov 01 15:52:53.981854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.981823 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f0400f5-aa65-4d7a-9f83-6235cbfa2b57" containerName="extract" Nov 01 15:52:53.984542 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:53.984527 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" Nov 01 15:52:54.000109 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.000090 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:52:54.000264 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.000218 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 15:52:54.001470 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.001449 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx"] Nov 01 15:52:54.080520 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.080492 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-nrqqk\"" Nov 01 15:52:54.162513 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.162477 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7fh\" (UniqueName: \"kubernetes.io/projected/91c82106-db3e-4fef-86b0-09e26b20781a-kube-api-access-ql7fh\") pod \"jobset-operator-674cd59dc5-gn9bx\" (UID: \"91c82106-db3e-4fef-86b0-09e26b20781a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" Nov 01 15:52:54.162675 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.162551 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c82106-db3e-4fef-86b0-09e26b20781a-tmp\") pod \"jobset-operator-674cd59dc5-gn9bx\" (UID: \"91c82106-db3e-4fef-86b0-09e26b20781a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" Nov 01 15:52:54.263882 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.263792 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c82106-db3e-4fef-86b0-09e26b20781a-tmp\") pod \"jobset-operator-674cd59dc5-gn9bx\" (UID: \"91c82106-db3e-4fef-86b0-09e26b20781a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" Nov 01 15:52:54.263882 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.263866 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7fh\" (UniqueName: \"kubernetes.io/projected/91c82106-db3e-4fef-86b0-09e26b20781a-kube-api-access-ql7fh\") pod \"jobset-operator-674cd59dc5-gn9bx\" (UID: \"91c82106-db3e-4fef-86b0-09e26b20781a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" Nov 01 15:52:54.264193 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.264171 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/91c82106-db3e-4fef-86b0-09e26b20781a-tmp\") pod \"jobset-operator-674cd59dc5-gn9bx\" (UID: \"91c82106-db3e-4fef-86b0-09e26b20781a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" Nov 01 15:52:54.276653 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.276622 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7fh\" (UniqueName: \"kubernetes.io/projected/91c82106-db3e-4fef-86b0-09e26b20781a-kube-api-access-ql7fh\") pod \"jobset-operator-674cd59dc5-gn9bx\" (UID: \"91c82106-db3e-4fef-86b0-09e26b20781a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" Nov 01 15:52:54.315595 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.315566 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" Nov 01 15:52:54.367340 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:52:54.367307 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:52:54.585702 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.585618 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx"] Nov 01 15:52:54.588964 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:52:54.588935 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91c82106_db3e_4fef_86b0_09e26b20781a.slice/crio-15a6bee1d03885051ba5f5cee6a43624da26212c0d067299e7605808b3c593a1 WatchSource:0}: Error finding container 15a6bee1d03885051ba5f5cee6a43624da26212c0d067299e7605808b3c593a1: Status 404 returned error can't find the container with id 15a6bee1d03885051ba5f5cee6a43624da26212c0d067299e7605808b3c593a1 Nov 01 15:52:54.590443 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.590428 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 15:52:54.657144 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:54.657115 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" event={"ID":"91c82106-db3e-4fef-86b0-09e26b20781a","Type":"ContainerStarted","Data":"15a6bee1d03885051ba5f5cee6a43624da26212c0d067299e7605808b3c593a1"} Nov 01 15:52:57.364654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:57.364612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:57.468783 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:57.468722 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:57.670170 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:57.670086 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" event={"ID":"91c82106-db3e-4fef-86b0-09e26b20781a","Type":"ContainerStarted","Data":"38a2321b7f26fccf9a0a79c71f19475f6fb623ebc867b3d46990a6a2720305aa"} Nov 01 15:52:57.688513 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:57.688467 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-gn9bx" podStartSLOduration=2.298613235 podStartE2EDuration="4.688454102s" totalImagesPullingTime="2.389840867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:52:57.688071285 +0000 UTC m=+315.391237223" watchObservedRunningTime="2026-11-01 15:52:57.688454102 +0000 UTC m=+315.391620037" Nov 01 15:52:58.531819 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.531786 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:58.688482 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.688448 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm"] Nov 01 15:52:58.692118 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.692097 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.696607 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.696587 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 15:52:58.696607 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.696586 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 15:52:58.696812 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.696801 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-4c5x6\"" Nov 01 15:52:58.697206 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.697193 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 15:52:58.697614 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.697597 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 15:52:58.697816 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.697791 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 15:52:58.738723 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.738696 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm"] Nov 01 15:52:58.799732 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.799660 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/542cbef7-5469-4ad4-b00b-8aacfbb5c111-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.799732 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.799719 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.799923 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.799839 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49nkf\" (UniqueName: \"kubernetes.io/projected/542cbef7-5469-4ad4-b00b-8aacfbb5c111-kube-api-access-49nkf\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.799923 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.799877 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.799923 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.799905 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/542cbef7-5469-4ad4-b00b-8aacfbb5c111-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.800031 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.800003 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.893569 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.893542 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:58.900780 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.900742 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.900917 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.900797 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49nkf\" (UniqueName: \"kubernetes.io/projected/542cbef7-5469-4ad4-b00b-8aacfbb5c111-kube-api-access-49nkf\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.900917 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.900821 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.900917 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.900845 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/542cbef7-5469-4ad4-b00b-8aacfbb5c111-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.901074 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.900955 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.901074 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.901011 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/542cbef7-5469-4ad4-b00b-8aacfbb5c111-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.901388 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.901366 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/542cbef7-5469-4ad4-b00b-8aacfbb5c111-manifests\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.903370 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.903346 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/542cbef7-5469-4ad4-b00b-8aacfbb5c111-tmp\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.903779 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.903694 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-metrics-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.903779 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.903694 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.903916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.903773 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/542cbef7-5469-4ad4-b00b-8aacfbb5c111-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.912930 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.912908 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49nkf\" (UniqueName: \"kubernetes.io/projected/542cbef7-5469-4ad4-b00b-8aacfbb5c111-kube-api-access-49nkf\") pod \"opendatahub-operator-controller-manager-6d5d696d67-7k6dm\" (UID: \"542cbef7-5469-4ad4-b00b-8aacfbb5c111\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:58.988185 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:58.988144 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:59.006263 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:59.006241 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:52:59.087463 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:59.087385 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:59.190039 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:59.189993 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:52:59.304138 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:59.304100 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm"] Nov 01 15:52:59.308565 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:52:59.308536 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod542cbef7_5469_4ad4_b00b_8aacfbb5c111.slice/crio-c2b5389671ca3e24d7cd36f35cfda29613e2b1c42fccef9af67f71556e12ad33 WatchSource:0}: Error finding container c2b5389671ca3e24d7cd36f35cfda29613e2b1c42fccef9af67f71556e12ad33: Status 404 returned error can't find the container with id c2b5389671ca3e24d7cd36f35cfda29613e2b1c42fccef9af67f71556e12ad33 Nov 01 15:52:59.679063 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:52:59.679032 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" event={"ID":"542cbef7-5469-4ad4-b00b-8aacfbb5c111","Type":"ContainerStarted","Data":"c2b5389671ca3e24d7cd36f35cfda29613e2b1c42fccef9af67f71556e12ad33"} Nov 01 15:53:01.246145 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:01.246089 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:01.347546 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:01.347485 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:02.564806 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:02.564666 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:02.669158 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:02.668965 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:02.693607 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:02.693581 4573 generic.go:356] "Generic (PLEG): container finished" podID="542cbef7-5469-4ad4-b00b-8aacfbb5c111" containerID="17065b16a4c43cec8c2110b6d545a20c747631689dd79712fa8478f099e7015e" exitCode=0 Nov 01 15:53:02.693702 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:02.693638 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" event={"ID":"542cbef7-5469-4ad4-b00b-8aacfbb5c111","Type":"ContainerDied","Data":"17065b16a4c43cec8c2110b6d545a20c747631689dd79712fa8478f099e7015e"} Nov 01 15:53:02.779802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:02.779765 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:02.882709 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:02.882680 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:03.700087 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:03.700055 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" event={"ID":"542cbef7-5469-4ad4-b00b-8aacfbb5c111","Type":"ContainerStarted","Data":"16a381fb4df51d98b6b9fda3185d578936a2b01934886c28c98455dae9c51dce"} Nov 01 15:53:03.700459 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:03.700179 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:53:03.735584 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:03.735537 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" podStartSLOduration=2.052996914 podStartE2EDuration="5.735524764s" totalImagesPullingTime="3.225066941s" totalInitContainerRuntime="457.460909ms" isStatefulPod=false podCreationTimestamp="2026-11-01 15:52:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:03.735321774 +0000 UTC m=+321.438487711" watchObservedRunningTime="2026-11-01 15:53:03.735524764 +0000 UTC m=+321.438690711" Nov 01 15:53:07.381544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:07.381505 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:07.482817 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:07.482780 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:12.549588 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:12.549559 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:13.553741 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:13.553703 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:13.654090 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:13.654059 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:14.706560 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:14.706529 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6d5d696d67-7k6dm" Nov 01 15:53:18.185723 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:18.185687 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:21.219455 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:21.219419 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:23.202111 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:23.202045 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:23.563004 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:23.562910 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:24.154958 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:24.154910 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:24.362669 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:53:24.362634 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:53:24.828214 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:24.828183 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:26.076306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:26.076266 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:26.187765 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:26.187713 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:27.561924 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:27.561875 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:29.795686 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:29.795651 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:30.608641 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:30.608600 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:30.711843 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:30.711807 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:31.688623 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.688586 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5"] Nov 01 15:53:31.693031 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.693008 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" Nov 01 15:53:31.696640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.696618 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 15:53:31.696640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.696618 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 15:53:31.697456 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.697439 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-4wbnr\"" Nov 01 15:53:31.703451 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.703429 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5"] Nov 01 15:53:31.762091 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.762056 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frpt\" (UniqueName: \"kubernetes.io/projected/be456d4e-da55-4307-8c9c-a14a4cb745d2-kube-api-access-4frpt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h52m5\" (UID: \"be456d4e-da55-4307-8c9c-a14a4cb745d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" Nov 01 15:53:31.762226 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.762112 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be456d4e-da55-4307-8c9c-a14a4cb745d2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h52m5\" (UID: \"be456d4e-da55-4307-8c9c-a14a4cb745d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" Nov 01 15:53:31.862525 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.862495 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be456d4e-da55-4307-8c9c-a14a4cb745d2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h52m5\" (UID: \"be456d4e-da55-4307-8c9c-a14a4cb745d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" Nov 01 15:53:31.862694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.862557 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frpt\" (UniqueName: \"kubernetes.io/projected/be456d4e-da55-4307-8c9c-a14a4cb745d2-kube-api-access-4frpt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h52m5\" (UID: \"be456d4e-da55-4307-8c9c-a14a4cb745d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" Nov 01 15:53:31.862934 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.862913 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be456d4e-da55-4307-8c9c-a14a4cb745d2-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h52m5\" (UID: \"be456d4e-da55-4307-8c9c-a14a4cb745d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" Nov 01 15:53:31.870121 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:31.870102 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frpt\" (UniqueName: \"kubernetes.io/projected/be456d4e-da55-4307-8c9c-a14a4cb745d2-kube-api-access-4frpt\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-h52m5\" (UID: \"be456d4e-da55-4307-8c9c-a14a4cb745d2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" Nov 01 15:53:32.004626 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:32.004549 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" Nov 01 15:53:32.141709 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:32.141665 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:32.241560 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:32.241527 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:32.282272 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:32.282205 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5"] Nov 01 15:53:32.285977 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:53:32.285948 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe456d4e_da55_4307_8c9c_a14a4cb745d2.slice/crio-1d751f8f08d5e9d7cd7dbaae75170c8c84f020d64c3130188d590e3767e22bc7 WatchSource:0}: Error finding container 1d751f8f08d5e9d7cd7dbaae75170c8c84f020d64c3130188d590e3767e22bc7: Status 404 returned error can't find the container with id 1d751f8f08d5e9d7cd7dbaae75170c8c84f020d64c3130188d590e3767e22bc7 Nov 01 15:53:32.363882 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:32.363856 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:32.465494 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:32.465459 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:32.816130 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:32.816069 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" event={"ID":"be456d4e-da55-4307-8c9c-a14a4cb745d2","Type":"ContainerStarted","Data":"1d751f8f08d5e9d7cd7dbaae75170c8c84f020d64c3130188d590e3767e22bc7"} Nov 01 15:53:33.000851 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:33.000816 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:33.182081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:33.182045 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:33.282022 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:33.281980 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:35.032955 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:35.032918 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:35.413988 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:35.413956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:35.830619 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:35.830586 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" event={"ID":"be456d4e-da55-4307-8c9c-a14a4cb745d2","Type":"ContainerStarted","Data":"d8d32ebddb09ac50ac762faf619d8dd627976fcf8917f028ce0aea2f78ff401f"} Nov 01 15:53:35.850819 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:35.850765 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-h52m5" podStartSLOduration=1.851548882 podStartE2EDuration="4.850735691s" totalImagesPullingTime="2.999186809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:35.849371387 +0000 UTC m=+353.552537323" watchObservedRunningTime="2026-11-01 15:53:35.850735691 +0000 UTC m=+353.553901626" Nov 01 15:53:36.227934 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.227896 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:36.358505 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.358473 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:36.373901 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.373871 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn"] Nov 01 15:53:36.377284 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.377263 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:36.390985 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.390961 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn"] Nov 01 15:53:36.405419 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.405399 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 15:53:36.405531 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.405484 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 15:53:36.502038 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.501960 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqk7\" (UniqueName: \"kubernetes.io/projected/09f03273-d479-473f-8a43-c1345ce65fe7-kube-api-access-bvqk7\") pod \"cert-manager-webhook-5dbbbf75cc-6dlmn\" (UID: \"09f03273-d479-473f-8a43-c1345ce65fe7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:36.502038 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.501992 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09f03273-d479-473f-8a43-c1345ce65fe7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6dlmn\" (UID: \"09f03273-d479-473f-8a43-c1345ce65fe7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:36.603541 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.603515 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqk7\" (UniqueName: \"kubernetes.io/projected/09f03273-d479-473f-8a43-c1345ce65fe7-kube-api-access-bvqk7\") pod \"cert-manager-webhook-5dbbbf75cc-6dlmn\" (UID: \"09f03273-d479-473f-8a43-c1345ce65fe7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:36.603638 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.603549 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09f03273-d479-473f-8a43-c1345ce65fe7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6dlmn\" (UID: \"09f03273-d479-473f-8a43-c1345ce65fe7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:36.612884 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.612857 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/09f03273-d479-473f-8a43-c1345ce65fe7-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6dlmn\" (UID: \"09f03273-d479-473f-8a43-c1345ce65fe7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:36.613202 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.613179 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqk7\" (UniqueName: \"kubernetes.io/projected/09f03273-d479-473f-8a43-c1345ce65fe7-kube-api-access-bvqk7\") pod \"cert-manager-webhook-5dbbbf75cc-6dlmn\" (UID: \"09f03273-d479-473f-8a43-c1345ce65fe7\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:36.657974 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.657945 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6"] Nov 01 15:53:36.661178 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.661163 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.674512 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.674320 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Nov 01 15:53:36.674668 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.674556 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-t6qrc\"" Nov 01 15:53:36.675329 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.675308 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 15:53:36.675444 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.675380 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Nov 01 15:53:36.675444 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.675418 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 15:53:36.676240 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.676223 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6"] Nov 01 15:53:36.688584 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.688269 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:36.698141 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.698115 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:36.704597 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.704571 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpr5n\" (UniqueName: \"kubernetes.io/projected/c0e421c1-a5b1-417a-a2a2-153403e5af56-kube-api-access-xpr5n\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.704689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.704605 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0e421c1-a5b1-417a-a2a2-153403e5af56-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.704735 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.704715 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e421c1-a5b1-417a-a2a2-153403e5af56-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.805795 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.805696 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e421c1-a5b1-417a-a2a2-153403e5af56-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.805941 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.805837 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpr5n\" (UniqueName: \"kubernetes.io/projected/c0e421c1-a5b1-417a-a2a2-153403e5af56-kube-api-access-xpr5n\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.805941 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.805866 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0e421c1-a5b1-417a-a2a2-153403e5af56-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.806574 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.806551 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c0e421c1-a5b1-417a-a2a2-153403e5af56-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.808167 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.808146 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0e421c1-a5b1-417a-a2a2-153403e5af56-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.839308 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.839285 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpr5n\" (UniqueName: \"kubernetes.io/projected/c0e421c1-a5b1-417a-a2a2-153403e5af56-kube-api-access-xpr5n\") pod \"kuadrant-console-plugin-66c4fb575d-4j9f6\" (UID: \"c0e421c1-a5b1-417a-a2a2-153403e5af56\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:36.890210 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.890180 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:36.972543 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:36.972513 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" Nov 01 15:53:37.029757 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.029722 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn"] Nov 01 15:53:37.034741 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:53:37.034699 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09f03273_d479_473f_8a43_c1345ce65fe7.slice/crio-9b6a3605f0abf561af55594a959fde00a6174cd9d7545674cb181bbfa1f4a064 WatchSource:0}: Error finding container 9b6a3605f0abf561af55594a959fde00a6174cd9d7545674cb181bbfa1f4a064: Status 404 returned error can't find the container with id 9b6a3605f0abf561af55594a959fde00a6174cd9d7545674cb181bbfa1f4a064 Nov 01 15:53:37.212678 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.212649 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:37.278006 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.271971 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6"] Nov 01 15:53:37.281281 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:53:37.281247 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0e421c1_a5b1_417a_a2a2_153403e5af56.slice/crio-949b7ff67e4fbe6a2937a99fbaaf3edc1e21df6d1c6f0b5d529e4512e38b874b WatchSource:0}: Error finding container 949b7ff67e4fbe6a2937a99fbaaf3edc1e21df6d1c6f0b5d529e4512e38b874b: Status 404 returned error can't find the container with id 949b7ff67e4fbe6a2937a99fbaaf3edc1e21df6d1c6f0b5d529e4512e38b874b Nov 01 15:53:37.363363 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.363314 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:37.631209 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.631172 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44"] Nov 01 15:53:37.636229 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.636208 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.642918 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.642888 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 15:53:37.645085 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.645066 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 15:53:37.648031 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.648008 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:37.648330 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.648311 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44"] Nov 01 15:53:37.666510 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.666487 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 15:53:37.666768 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.666738 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 01 15:53:37.667537 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.667516 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-89t2n\"" Nov 01 15:53:37.713638 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.713598 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbjkc\" (UniqueName: \"kubernetes.io/projected/ce438594-887f-464e-b710-3502d00c4ead-kube-api-access-tbjkc\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.713825 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.713660 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce438594-887f-464e-b710-3502d00c4ead-tls-certs\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.713825 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.713686 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce438594-887f-464e-b710-3502d00c4ead-tmp\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.773697 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.773668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:37.814219 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.814185 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbjkc\" (UniqueName: \"kubernetes.io/projected/ce438594-887f-464e-b710-3502d00c4ead-kube-api-access-tbjkc\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.814379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.814231 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce438594-887f-464e-b710-3502d00c4ead-tls-certs\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.814379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.814249 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce438594-887f-464e-b710-3502d00c4ead-tmp\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.816528 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.816498 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce438594-887f-464e-b710-3502d00c4ead-tmp\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.816822 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.816803 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce438594-887f-464e-b710-3502d00c4ead-tls-certs\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.830272 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.830246 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbjkc\" (UniqueName: \"kubernetes.io/projected/ce438594-887f-464e-b710-3502d00c4ead-kube-api-access-tbjkc\") pod \"kube-auth-proxy-6759f4f87c-t6l44\" (UID: \"ce438594-887f-464e-b710-3502d00c4ead\") " pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:37.839945 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.839916 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" event={"ID":"c0e421c1-a5b1-417a-a2a2-153403e5af56","Type":"ContainerStarted","Data":"949b7ff67e4fbe6a2937a99fbaaf3edc1e21df6d1c6f0b5d529e4512e38b874b"} Nov 01 15:53:37.841033 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.841011 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" event={"ID":"09f03273-d479-473f-8a43-c1345ce65fe7","Type":"ContainerStarted","Data":"9b6a3605f0abf561af55594a959fde00a6174cd9d7545674cb181bbfa1f4a064"} Nov 01 15:53:37.899143 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.899069 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:37.950269 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:37.950239 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" Nov 01 15:53:38.089134 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.089093 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:38.199925 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.199835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:38.243530 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.243501 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44"] Nov 01 15:53:38.245582 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:53:38.245555 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce438594_887f_464e_b710_3502d00c4ead.slice/crio-add9e3a338d886b82fe16008c76645de091e75739267eeecb6b1c106837d13b0 WatchSource:0}: Error finding container add9e3a338d886b82fe16008c76645de091e75739267eeecb6b1c106837d13b0: Status 404 returned error can't find the container with id add9e3a338d886b82fe16008c76645de091e75739267eeecb6b1c106837d13b0 Nov 01 15:53:38.626340 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.626312 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-crfw6"] Nov 01 15:53:38.630324 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.630072 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:38.637728 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.637492 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 15:53:38.641824 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.641234 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-crfw6"] Nov 01 15:53:38.641824 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.641550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:38.725681 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.725500 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80002057-c5b5-4472-b67e-6a29c51a2db6-config-file\") pod \"limitador-limitador-74dfcfb894-crfw6\" (UID: \"80002057-c5b5-4472-b67e-6a29c51a2db6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:38.725681 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.725579 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkhn4\" (UniqueName: \"kubernetes.io/projected/80002057-c5b5-4472-b67e-6a29c51a2db6-kube-api-access-lkhn4\") pod \"limitador-limitador-74dfcfb894-crfw6\" (UID: \"80002057-c5b5-4472-b67e-6a29c51a2db6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:38.730197 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.730168 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-crfw6"] Nov 01 15:53:38.731597 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.731526 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:38.826207 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.826168 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkhn4\" (UniqueName: \"kubernetes.io/projected/80002057-c5b5-4472-b67e-6a29c51a2db6-kube-api-access-lkhn4\") pod \"limitador-limitador-74dfcfb894-crfw6\" (UID: \"80002057-c5b5-4472-b67e-6a29c51a2db6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:38.826392 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.826277 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80002057-c5b5-4472-b67e-6a29c51a2db6-config-file\") pod \"limitador-limitador-74dfcfb894-crfw6\" (UID: \"80002057-c5b5-4472-b67e-6a29c51a2db6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:38.827079 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.827054 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80002057-c5b5-4472-b67e-6a29c51a2db6-config-file\") pod \"limitador-limitador-74dfcfb894-crfw6\" (UID: \"80002057-c5b5-4472-b67e-6a29c51a2db6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:38.827908 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.827879 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:38.845100 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.845073 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkhn4\" (UniqueName: \"kubernetes.io/projected/80002057-c5b5-4472-b67e-6a29c51a2db6-kube-api-access-lkhn4\") pod \"limitador-limitador-74dfcfb894-crfw6\" (UID: \"80002057-c5b5-4472-b67e-6a29c51a2db6\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:38.849893 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.849823 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" event={"ID":"ce438594-887f-464e-b710-3502d00c4ead","Type":"ContainerStarted","Data":"add9e3a338d886b82fe16008c76645de091e75739267eeecb6b1c106837d13b0"} Nov 01 15:53:38.961850 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:38.961647 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:39.272303 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.272145 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-crfw6"] Nov 01 15:53:39.278142 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:53:39.278037 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80002057_c5b5_4472_b67e_6a29c51a2db6.slice/crio-6d2a01d10570c9c175d38362ccd89481c8d3036520db253dcbba1a7d65436294 WatchSource:0}: Error finding container 6d2a01d10570c9c175d38362ccd89481c8d3036520db253dcbba1a7d65436294: Status 404 returned error can't find the container with id 6d2a01d10570c9c175d38362ccd89481c8d3036520db253dcbba1a7d65436294 Nov 01 15:53:39.411874 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.410623 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:53:39.415876 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.414131 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:39.423403 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.423379 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:53:39.455970 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.455942 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:53:39.535872 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.535789 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18b562d7-dad3-49f7-aaf0-49c9a465f7f9-config-file\") pod \"limitador-limitador-7c9b87d574-gxdck\" (UID: \"18b562d7-dad3-49f7-aaf0-49c9a465f7f9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:39.535872 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.535851 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkj6\" (UniqueName: \"kubernetes.io/projected/18b562d7-dad3-49f7-aaf0-49c9a465f7f9-kube-api-access-cbkj6\") pod \"limitador-limitador-7c9b87d574-gxdck\" (UID: \"18b562d7-dad3-49f7-aaf0-49c9a465f7f9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:39.558968 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.558937 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:39.638040 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.636800 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18b562d7-dad3-49f7-aaf0-49c9a465f7f9-config-file\") pod \"limitador-limitador-7c9b87d574-gxdck\" (UID: \"18b562d7-dad3-49f7-aaf0-49c9a465f7f9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:39.638040 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.636872 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkj6\" (UniqueName: \"kubernetes.io/projected/18b562d7-dad3-49f7-aaf0-49c9a465f7f9-kube-api-access-cbkj6\") pod \"limitador-limitador-7c9b87d574-gxdck\" (UID: \"18b562d7-dad3-49f7-aaf0-49c9a465f7f9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:39.638040 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.637982 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/18b562d7-dad3-49f7-aaf0-49c9a465f7f9-config-file\") pod \"limitador-limitador-7c9b87d574-gxdck\" (UID: \"18b562d7-dad3-49f7-aaf0-49c9a465f7f9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:39.647092 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.647052 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkj6\" (UniqueName: \"kubernetes.io/projected/18b562d7-dad3-49f7-aaf0-49c9a465f7f9-kube-api-access-cbkj6\") pod \"limitador-limitador-7c9b87d574-gxdck\" (UID: \"18b562d7-dad3-49f7-aaf0-49c9a465f7f9\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:39.664086 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.664019 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:39.738984 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.738947 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:39.765019 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.764973 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:39.865158 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:39.865122 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" event={"ID":"80002057-c5b5-4472-b67e-6a29c51a2db6","Type":"ContainerStarted","Data":"6d2a01d10570c9c175d38362ccd89481c8d3036520db253dcbba1a7d65436294"} Nov 01 15:53:40.049544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:40.048680 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:53:40.053539 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:53:40.053505 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18b562d7_dad3_49f7_aaf0_49c9a465f7f9.slice/crio-4a2a5bd5619cc75819f2ab2042ad102ad5fdc864586bcca133da2634f146eed8 WatchSource:0}: Error finding container 4a2a5bd5619cc75819f2ab2042ad102ad5fdc864586bcca133da2634f146eed8: Status 404 returned error can't find the container with id 4a2a5bd5619cc75819f2ab2042ad102ad5fdc864586bcca133da2634f146eed8 Nov 01 15:53:40.874899 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:40.874814 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" event={"ID":"18b562d7-dad3-49f7-aaf0-49c9a465f7f9","Type":"ContainerStarted","Data":"4a2a5bd5619cc75819f2ab2042ad102ad5fdc864586bcca133da2634f146eed8"} Nov 01 15:53:42.359454 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:42.359424 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:42.462838 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:42.462731 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:42.771959 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:42.771933 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 01 15:53:42.896562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:42.895895 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" event={"ID":"ce438594-887f-464e-b710-3502d00c4ead","Type":"ContainerStarted","Data":"999e524cb7ebc0371bd43ec3f21847edb6989d29a8fe06abfc5714ea29d2f0ab"} Nov 01 15:53:42.898576 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:42.898546 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" event={"ID":"09f03273-d479-473f-8a43-c1345ce65fe7","Type":"ContainerStarted","Data":"8b4095baba773220f3a37833337d3575128b465f69ead32e095bd6df95ca2002"} Nov 01 15:53:42.898943 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:42.898914 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:42.917643 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:42.917532 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6759f4f87c-t6l44" podStartSLOduration=1.398406927 podStartE2EDuration="5.917512813s" totalImagesPullingTime="4.519105886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:42.913464822 +0000 UTC m=+360.616630759" watchObservedRunningTime="2026-11-01 15:53:42.917512813 +0000 UTC m=+360.620678753" Nov 01 15:53:42.936855 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:42.936793 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" podStartSLOduration=1.86132556 podStartE2EDuration="6.936774515s" totalImagesPullingTime="5.075448955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:42.935435497 +0000 UTC m=+360.638601436" watchObservedRunningTime="2026-11-01 15:53:42.936774515 +0000 UTC m=+360.639940443" Nov 01 15:53:43.572012 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:43.571983 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:43.673494 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:43.673411 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:43.906530 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:43.906414 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" event={"ID":"80002057-c5b5-4472-b67e-6a29c51a2db6","Type":"ContainerStarted","Data":"c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f"} Nov 01 15:53:43.906530 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:43.906493 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:43.908321 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:43.908294 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" event={"ID":"18b562d7-dad3-49f7-aaf0-49c9a465f7f9","Type":"ContainerStarted","Data":"4d0119f5039e5348af136c64b9d6c25f2ae47bb6b7d5641040672538838d96e7"} Nov 01 15:53:43.908646 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:43.908621 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:43.938199 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:43.938083 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" podStartSLOduration=1.799639123 podStartE2EDuration="5.938066626s" totalImagesPullingTime="4.138427503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:43.937911101 +0000 UTC m=+361.641077039" watchObservedRunningTime="2026-11-01 15:53:43.938066626 +0000 UTC m=+361.641232563" Nov 01 15:53:43.965445 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:43.965376 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" podStartSLOduration=1.498606526 podStartE2EDuration="4.965357512s" totalImagesPullingTime="3.466750986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:43.963426727 +0000 UTC m=+361.666592664" watchObservedRunningTime="2026-11-01 15:53:43.965357512 +0000 UTC m=+361.668523448" Nov 01 15:53:44.126677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:44.126643 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:44.231669 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:44.231597 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:44.918517 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:44.918476 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" event={"ID":"c0e421c1-a5b1-417a-a2a2-153403e5af56","Type":"ContainerStarted","Data":"3228988c7c1590a991ee688a071ce852e761e1835a7047235f97f360fb548ef1"} Nov 01 15:53:47.032284 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:47.032251 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:48.911226 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:48.911195 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" Nov 01 15:53:48.928668 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:48.928618 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-4j9f6" podStartSLOduration=5.432977821 podStartE2EDuration="12.928603786s" totalImagesPullingTime="7.495625965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:44.933543418 +0000 UTC m=+362.636709393" watchObservedRunningTime="2026-11-01 15:53:48.928603786 +0000 UTC m=+366.631769722" Nov 01 15:53:49.773439 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:49.773402 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:49.872884 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:49.872846 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:51.160646 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:51.160611 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:51.261105 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:51.261079 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:51.837891 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:51.837854 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:51.908004 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:51.907972 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:54.363487 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:53:54.363446 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:53:54.919711 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:54.919683 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:55.923268 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:55.923232 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-gxdck" Nov 01 15:53:55.966965 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:55.966930 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-crfw6"] Nov 01 15:53:55.967170 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:55.967134 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" podUID="80002057-c5b5-4472-b67e-6a29c51a2db6" containerName="limitador" containerID="cri-o://c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f" gracePeriod=30 Nov 01 15:53:56.500555 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.500531 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:56.511957 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.511937 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80002057-c5b5-4472-b67e-6a29c51a2db6-kube-api-access-lkhn4\" (UniqueName: \"kubernetes.io/projected/80002057-c5b5-4472-b67e-6a29c51a2db6-kube-api-access-lkhn4\") pod \"80002057-c5b5-4472-b67e-6a29c51a2db6\" (UID: \"80002057-c5b5-4472-b67e-6a29c51a2db6\") " Nov 01 15:53:56.512055 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.511986 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80002057-c5b5-4472-b67e-6a29c51a2db6-config-file\" (UniqueName: \"kubernetes.io/configmap/80002057-c5b5-4472-b67e-6a29c51a2db6-config-file\") pod \"80002057-c5b5-4472-b67e-6a29c51a2db6\" (UID: \"80002057-c5b5-4472-b67e-6a29c51a2db6\") " Nov 01 15:53:56.512309 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.512290 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80002057-c5b5-4472-b67e-6a29c51a2db6-config-file" pod "80002057-c5b5-4472-b67e-6a29c51a2db6" (UID: "80002057-c5b5-4472-b67e-6a29c51a2db6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 15:53:56.514031 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.514006 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80002057-c5b5-4472-b67e-6a29c51a2db6-kube-api-access-lkhn4" pod "80002057-c5b5-4472-b67e-6a29c51a2db6" (UID: "80002057-c5b5-4472-b67e-6a29c51a2db6"). InnerVolumeSpecName "kube-api-access-lkhn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:53:56.613003 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.612976 4573 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/80002057-c5b5-4472-b67e-6a29c51a2db6-config-file\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:56.613003 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.613003 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkhn4\" (UniqueName: \"kubernetes.io/projected/80002057-c5b5-4472-b67e-6a29c51a2db6-kube-api-access-lkhn4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:53:56.826659 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.826619 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-vmz2v"] Nov 01 15:53:56.827022 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.827007 4573 cpu_manager.go:414] "containerMap: removing container" podUID="80002057-c5b5-4472-b67e-6a29c51a2db6" containerName="limitador" Nov 01 15:53:56.827077 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.827025 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80002057-c5b5-4472-b67e-6a29c51a2db6" containerName="limitador" Nov 01 15:53:56.827113 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.827094 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="80002057-c5b5-4472-b67e-6a29c51a2db6" containerName="limitador" Nov 01 15:53:56.830084 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.830069 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vmz2v" Nov 01 15:53:56.836236 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.836191 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-8brrk\"" Nov 01 15:53:56.838666 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.838644 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vmz2v"] Nov 01 15:53:56.915288 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.915218 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea-bound-sa-token\") pod \"cert-manager-597b95cb84-vmz2v\" (UID: \"8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea\") " pod="cert-manager/cert-manager-597b95cb84-vmz2v" Nov 01 15:53:56.915288 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.915250 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzjfn\" (UniqueName: \"kubernetes.io/projected/8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea-kube-api-access-bzjfn\") pod \"cert-manager-597b95cb84-vmz2v\" (UID: \"8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea\") " pod="cert-manager/cert-manager-597b95cb84-vmz2v" Nov 01 15:53:56.970185 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.970153 4573 generic.go:356] "Generic (PLEG): container finished" podID="80002057-c5b5-4472-b67e-6a29c51a2db6" containerID="c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f" exitCode=0 Nov 01 15:53:56.970562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.970199 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" Nov 01 15:53:56.970562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.970222 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" event={"ID":"80002057-c5b5-4472-b67e-6a29c51a2db6","Type":"ContainerDied","Data":"c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f"} Nov 01 15:53:56.970562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.970266 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-crfw6" event={"ID":"80002057-c5b5-4472-b67e-6a29c51a2db6","Type":"ContainerDied","Data":"6d2a01d10570c9c175d38362ccd89481c8d3036520db253dcbba1a7d65436294"} Nov 01 15:53:56.970562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.970290 4573 scope.go:122] "RemoveContainer" containerID="c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f" Nov 01 15:53:56.979162 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.979143 4573 scope.go:122] "RemoveContainer" containerID="c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f" Nov 01 15:53:56.979424 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:53:56.979405 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f\": container with ID starting with c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f not found: ID does not exist" containerID="c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f" Nov 01 15:53:56.979493 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.979434 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f"} err="failed to get container status \"c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f\": rpc error: code = NotFound desc = could not find container \"c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f\": container with ID starting with c832cd69df308c4e834887ecafcd1c5a377a84744ec9031f14ca9e30dc1a5c3f not found: ID does not exist" Nov 01 15:53:56.991245 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.991223 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-crfw6"] Nov 01 15:53:56.993485 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:56.993467 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-crfw6"] Nov 01 15:53:57.016501 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.016482 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea-bound-sa-token\") pod \"cert-manager-597b95cb84-vmz2v\" (UID: \"8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea\") " pod="cert-manager/cert-manager-597b95cb84-vmz2v" Nov 01 15:53:57.016590 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.016511 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzjfn\" (UniqueName: \"kubernetes.io/projected/8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea-kube-api-access-bzjfn\") pod \"cert-manager-597b95cb84-vmz2v\" (UID: \"8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea\") " pod="cert-manager/cert-manager-597b95cb84-vmz2v" Nov 01 15:53:57.025364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.025339 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzjfn\" (UniqueName: \"kubernetes.io/projected/8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea-kube-api-access-bzjfn\") pod \"cert-manager-597b95cb84-vmz2v\" (UID: \"8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea\") " pod="cert-manager/cert-manager-597b95cb84-vmz2v" Nov 01 15:53:57.028957 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.028934 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea-bound-sa-token\") pod \"cert-manager-597b95cb84-vmz2v\" (UID: \"8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea\") " pod="cert-manager/cert-manager-597b95cb84-vmz2v" Nov 01 15:53:57.142129 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.142095 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-vmz2v" Nov 01 15:53:57.414081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.414059 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-vmz2v"] Nov 01 15:53:57.415982 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:53:57.415949 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dd37c92_1df8_4d3d_9f2f_b9fa0cbf4aea.slice/crio-cce80c6545e673cf72f45a7f6f3c65449049db55d981358f9d391002ddec0e0d WatchSource:0}: Error finding container cce80c6545e673cf72f45a7f6f3c65449049db55d981358f9d391002ddec0e0d: Status 404 returned error can't find the container with id cce80c6545e673cf72f45a7f6f3c65449049db55d981358f9d391002ddec0e0d Nov 01 15:53:57.977075 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.977038 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vmz2v" event={"ID":"8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea","Type":"ContainerStarted","Data":"ff61de6e765ed9ad0662a9417f871f4add508f8a77a7322102a019dcbda0dbc1"} Nov 01 15:53:57.977075 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.977082 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-vmz2v" event={"ID":"8dd37c92-1df8-4d3d-9f2f-b9fa0cbf4aea","Type":"ContainerStarted","Data":"cce80c6545e673cf72f45a7f6f3c65449049db55d981358f9d391002ddec0e0d"} Nov 01 15:53:57.993683 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:57.993635 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-vmz2v" podStartSLOduration=1.99362246 podStartE2EDuration="1.99362246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:53:57.992108119 +0000 UTC m=+375.695274056" watchObservedRunningTime="2026-11-01 15:53:57.99362246 +0000 UTC m=+375.696788396" Nov 01 15:53:58.455676 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:58.455634 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80002057-c5b5-4472-b67e-6a29c51a2db6" path="/var/lib/kubelet/pods/80002057-c5b5-4472-b67e-6a29c51a2db6/volumes" Nov 01 15:53:58.756283 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:58.756199 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:58.856458 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:58.856426 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:59.560055 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:59.560017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:59.638584 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:59.638553 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:59.805179 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:59.805150 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:53:59.904431 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:53:59.904392 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:00.488884 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.488848 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq"] Nov 01 15:54:00.492082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.492060 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:00.499167 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.499142 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 15:54:00.499607 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.499574 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 15:54:00.500131 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.500111 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq"] Nov 01 15:54:00.540179 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.540155 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 15:54:00.540324 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.540259 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-bjsvn\"" Nov 01 15:54:00.547613 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.547592 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9417c68-46f1-441f-a77b-c7943426bbd7-data\") pod \"postgres-7cdbf6dc44-9t8zq\" (UID: \"c9417c68-46f1-441f-a77b-c7943426bbd7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:00.547667 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.547643 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfk5\" (UniqueName: \"kubernetes.io/projected/c9417c68-46f1-441f-a77b-c7943426bbd7-kube-api-access-9pfk5\") pod \"postgres-7cdbf6dc44-9t8zq\" (UID: \"c9417c68-46f1-441f-a77b-c7943426bbd7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:00.570506 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.570480 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:00.648700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.648676 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9417c68-46f1-441f-a77b-c7943426bbd7-data\") pod \"postgres-7cdbf6dc44-9t8zq\" (UID: \"c9417c68-46f1-441f-a77b-c7943426bbd7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:00.648832 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.648715 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfk5\" (UniqueName: \"kubernetes.io/projected/c9417c68-46f1-441f-a77b-c7943426bbd7-kube-api-access-9pfk5\") pod \"postgres-7cdbf6dc44-9t8zq\" (UID: \"c9417c68-46f1-441f-a77b-c7943426bbd7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:00.649080 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.649061 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c9417c68-46f1-441f-a77b-c7943426bbd7-data\") pod \"postgres-7cdbf6dc44-9t8zq\" (UID: \"c9417c68-46f1-441f-a77b-c7943426bbd7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:00.656601 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.656584 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfk5\" (UniqueName: \"kubernetes.io/projected/c9417c68-46f1-441f-a77b-c7943426bbd7-kube-api-access-9pfk5\") pod \"postgres-7cdbf6dc44-9t8zq\" (UID: \"c9417c68-46f1-441f-a77b-c7943426bbd7\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:00.805391 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:00.805318 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:01.080101 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:01.080080 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq"] Nov 01 15:54:01.082195 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:54:01.082170 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9417c68_46f1_441f_a77b_c7943426bbd7.slice/crio-f5134943ea84456153140d6efc8c761c6e9c3c8b38a09ad967260f64bfa4464d WatchSource:0}: Error finding container f5134943ea84456153140d6efc8c761c6e9c3c8b38a09ad967260f64bfa4464d: Status 404 returned error can't find the container with id f5134943ea84456153140d6efc8c761c6e9c3c8b38a09ad967260f64bfa4464d Nov 01 15:54:01.994798 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:01.994763 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" event={"ID":"c9417c68-46f1-441f-a77b-c7943426bbd7","Type":"ContainerStarted","Data":"f5134943ea84456153140d6efc8c761c6e9c3c8b38a09ad967260f64bfa4464d"} Nov 01 15:54:04.361106 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:04.361070 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:04.460300 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:04.460266 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:07.024204 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:07.024171 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" event={"ID":"c9417c68-46f1-441f-a77b-c7943426bbd7","Type":"ContainerStarted","Data":"25bfd9c784c5b01292a930effd2d38679e8a9270ed7294545132cfb813858242"} Nov 01 15:54:07.024654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:07.024320 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:07.039860 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:07.039783 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" podStartSLOduration=1.3395027640000001 podStartE2EDuration="7.039770604s" totalImagesPullingTime="5.70026784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:07.039084586 +0000 UTC m=+384.742250522" watchObservedRunningTime="2026-11-01 15:54:07.039770604 +0000 UTC m=+384.742936533" Nov 01 15:54:07.184663 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:07.184630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:07.285764 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:07.285729 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:11.934379 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:11.934326 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:12.035254 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:12.035222 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:13.058115 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:13.058086 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-9t8zq" Nov 01 15:54:18.609367 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:18.609331 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:18.677127 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:18.677094 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:19.751277 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:19.751228 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:19.844402 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:19.844372 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:19.917939 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:19.917909 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:20.017903 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:20.017827 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:20.121715 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:20.121683 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:20.388586 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:20.388539 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:20.548340 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:20.548310 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:22.786934 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:22.786902 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:22.887390 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:22.887363 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:23.670310 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:23.670230 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:23.769648 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:23.769618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:24.363080 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:54:24.363040 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:54:25.678688 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:25.678647 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:25.811821 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:25.811785 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:26.865268 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:26.865219 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:27.204846 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:27.204745 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:27.306692 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:27.306655 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:29.842060 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:29.842021 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:31.800304 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:31.800257 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:31.894940 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:31.894908 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:32.334802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.334731 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:32.440653 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.440608 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:32.537571 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.537538 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-wh7xc"] Nov 01 15:54:32.541000 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.540978 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.544608 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.544589 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 15:54:32.545406 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.545388 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 15:54:32.555081 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.555059 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-wh7xc"] Nov 01 15:54:32.566793 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.566765 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:32.582561 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.582538 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 01 15:54:32.582682 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.582629 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-dpr7v\"" Nov 01 15:54:32.582727 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.582698 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 01 15:54:32.641680 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.641650 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89878e6b-ba80-4311-a306-ba78b713d080-cert\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.641891 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.641687 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89878e6b-ba80-4311-a306-ba78b713d080-metrics-certs\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.641891 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.641776 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2jjr\" (UniqueName: \"kubernetes.io/projected/89878e6b-ba80-4311-a306-ba78b713d080-kube-api-access-x2jjr\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.662899 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.662847 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:32.742914 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.742885 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89878e6b-ba80-4311-a306-ba78b713d080-metrics-certs\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.743047 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.742945 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2jjr\" (UniqueName: \"kubernetes.io/projected/89878e6b-ba80-4311-a306-ba78b713d080-kube-api-access-x2jjr\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.743047 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.742999 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89878e6b-ba80-4311-a306-ba78b713d080-cert\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.745357 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.745331 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89878e6b-ba80-4311-a306-ba78b713d080-metrics-certs\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.745462 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.745371 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89878e6b-ba80-4311-a306-ba78b713d080-cert\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.757715 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.757693 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2jjr\" (UniqueName: \"kubernetes.io/projected/89878e6b-ba80-4311-a306-ba78b713d080-kube-api-access-x2jjr\") pod \"odh-model-controller-769875db56-wh7xc\" (UID: \"89878e6b-ba80-4311-a306-ba78b713d080\") " pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:32.762134 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.762114 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:32.853724 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:32.853694 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:33.179121 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:33.179096 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-wh7xc"] Nov 01 15:54:33.180205 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:54:33.180176 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89878e6b_ba80_4311_a306_ba78b713d080.slice/crio-dcb78cb0301f439065127337972747aa0ecd2274937440e45f5ff98350e2d654 WatchSource:0}: Error finding container dcb78cb0301f439065127337972747aa0ecd2274937440e45f5ff98350e2d654: Status 404 returned error can't find the container with id dcb78cb0301f439065127337972747aa0ecd2274937440e45f5ff98350e2d654 Nov 01 15:54:33.581689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:33.581616 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:33.681771 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:33.681724 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:33.777727 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:33.777693 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:33.878897 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:33.878861 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:34.144776 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:34.144647 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-wh7xc" event={"ID":"89878e6b-ba80-4311-a306-ba78b713d080","Type":"ContainerStarted","Data":"dcb78cb0301f439065127337972747aa0ecd2274937440e45f5ff98350e2d654"} Nov 01 15:54:36.156608 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:36.156581 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-wh7xc" event={"ID":"89878e6b-ba80-4311-a306-ba78b713d080","Type":"ContainerStarted","Data":"2599db011018e16cce34c98f720664e922cd5b8c5e6c4c6e35537e6bbc85bf23"} Nov 01 15:54:36.156999 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:36.156725 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:36.177993 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:36.177931 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-wh7xc" podStartSLOduration=1.281310388 podStartE2EDuration="4.177917345s" totalImagesPullingTime="2.896606957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:36.177268463 +0000 UTC m=+413.880434399" watchObservedRunningTime="2026-11-01 15:54:36.177917345 +0000 UTC m=+413.881083282" Nov 01 15:54:38.683461 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:38.683423 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:38.783639 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:38.783599 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:39.232993 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.232964 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg"] Nov 01 15:54:39.252184 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.252155 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg"] Nov 01 15:54:39.252359 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.252243 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.256103 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.256069 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 01 15:54:39.257054 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.256982 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 01 15:54:39.257054 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.257020 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-fn28c\"" Nov 01 15:54:39.257196 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.257058 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 01 15:54:39.302075 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.302042 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d336c6d4-1bfc-4234-a0f0-46285b649e3e-cert\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.302075 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.302076 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc54f\" (UniqueName: \"kubernetes.io/projected/d336c6d4-1bfc-4234-a0f0-46285b649e3e-kube-api-access-rc54f\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.302244 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.302169 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d336c6d4-1bfc-4234-a0f0-46285b649e3e-manager-config\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.302244 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.302213 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d336c6d4-1bfc-4234-a0f0-46285b649e3e-metrics-certs\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.402709 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.402677 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d336c6d4-1bfc-4234-a0f0-46285b649e3e-cert\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.402709 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.402711 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc54f\" (UniqueName: \"kubernetes.io/projected/d336c6d4-1bfc-4234-a0f0-46285b649e3e-kube-api-access-rc54f\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.402932 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.402778 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d336c6d4-1bfc-4234-a0f0-46285b649e3e-manager-config\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.402999 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.402934 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d336c6d4-1bfc-4234-a0f0-46285b649e3e-metrics-certs\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.403444 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.403423 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d336c6d4-1bfc-4234-a0f0-46285b649e3e-manager-config\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.405204 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.405182 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d336c6d4-1bfc-4234-a0f0-46285b649e3e-cert\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.405284 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.405222 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d336c6d4-1bfc-4234-a0f0-46285b649e3e-metrics-certs\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.410353 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.410331 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc54f\" (UniqueName: \"kubernetes.io/projected/d336c6d4-1bfc-4234-a0f0-46285b649e3e-kube-api-access-rc54f\") pod \"jobset-controller-manager-656fd89bb5-zlmkg\" (UID: \"d336c6d4-1bfc-4234-a0f0-46285b649e3e\") " pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.564325 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.564251 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:39.844228 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:39.844198 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg"] Nov 01 15:54:39.846503 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:54:39.846471 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd336c6d4_1bfc_4234_a0f0_46285b649e3e.slice/crio-8b265068f66695bed2dfaf15318f250b8f82a9fdfb8e0afcca1803793f7a9ac2 WatchSource:0}: Error finding container 8b265068f66695bed2dfaf15318f250b8f82a9fdfb8e0afcca1803793f7a9ac2: Status 404 returned error can't find the container with id 8b265068f66695bed2dfaf15318f250b8f82a9fdfb8e0afcca1803793f7a9ac2 Nov 01 15:54:40.176615 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:40.176578 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" event={"ID":"d336c6d4-1bfc-4234-a0f0-46285b649e3e","Type":"ContainerStarted","Data":"8b265068f66695bed2dfaf15318f250b8f82a9fdfb8e0afcca1803793f7a9ac2"} Nov 01 15:54:44.196156 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:44.196117 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" event={"ID":"d336c6d4-1bfc-4234-a0f0-46285b649e3e","Type":"ContainerStarted","Data":"8d2649808f0123c71eaa709504172a747b543a45f7eb20025ae7acf5fc36e594"} Nov 01 15:54:44.196653 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:44.196231 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:44.212721 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:44.212672 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" podStartSLOduration=1.461106046 podStartE2EDuration="5.212660113s" totalImagesPullingTime="3.751554067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:54:44.211917818 +0000 UTC m=+421.915083768" watchObservedRunningTime="2026-11-01 15:54:44.212660113 +0000 UTC m=+421.915826048" Nov 01 15:54:47.164134 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:47.164102 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-wh7xc" Nov 01 15:54:53.971726 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:53.971623 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:54.069012 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:54.068979 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:54.363303 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:54:54.363266 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:54:55.207297 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:55.207263 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-656fd89bb5-zlmkg" Nov 01 15:54:55.260427 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:55.260390 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:55.362554 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:55.362516 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:55.562249 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:55.562171 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:55.668665 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:55.668636 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:54:56.029992 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:56.029959 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:54:58.205836 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:54:58.205744 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:04.696362 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:55:04.696330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:04.797593 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:55:04.797562 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:24.363533 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:55:24.363492 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:55:29.883218 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:55:29.883180 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:55:54.362916 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:55:54.362838 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:55:59.750462 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:55:59.750426 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:02.385302 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:02.385261 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:02.482688 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:02.482653 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:20.600537 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:20.600502 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:21.094482 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:21.094446 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:21.441144 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:21.441113 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:56:22.405463 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:22.405430 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:22.930699 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:22.930668 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:56:24.363730 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:56:24.363694 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:56:24.593975 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:24.593939 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:25.171553 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:25.171523 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:56:26.129725 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:26.129689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:26.840356 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:26.840322 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:56:27.881118 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.881085 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt"] Nov 01 15:56:27.884679 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.884656 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:27.894488 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.894462 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ffz56\"" Nov 01 15:56:27.894700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.894677 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 01 15:56:27.896271 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.896247 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt"] Nov 01 15:56:27.919409 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.919384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:27.936621 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.936595 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 01 15:56:27.936621 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.936592 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 01 15:56:27.953418 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.953391 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8mt\" (UniqueName: \"kubernetes.io/projected/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-kube-api-access-8l8mt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:27.953513 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.953423 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:27.953513 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.953453 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:27.953609 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.953589 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:27.953648 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.953621 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:27.953648 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:27.953638 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055056 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055024 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055209 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055064 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055209 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055081 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055209 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055178 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8mt\" (UniqueName: \"kubernetes.io/projected/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-kube-api-access-8l8mt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055392 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055208 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055392 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055245 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055485 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055729 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055607 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.055729 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.055650 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.057335 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.057311 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.057580 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.057564 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.063174 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.063152 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8mt\" (UniqueName: \"kubernetes.io/projected/a0218c49-8d6d-4bb3-8cf7-10634e6c7a44-kube-api-access-8l8mt\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt\" (UID: \"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.197598 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.197520 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:28.476626 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.476552 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt"] Nov 01 15:56:28.479589 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:56:28.479557 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0218c49_8d6d_4bb3_8cf7_10634e6c7a44.slice/crio-f6eb252d4fb3131753127d810b34f8017a67e30b349800aaa4c75193cfba411b WatchSource:0}: Error finding container f6eb252d4fb3131753127d810b34f8017a67e30b349800aaa4c75193cfba411b: Status 404 returned error can't find the container with id f6eb252d4fb3131753127d810b34f8017a67e30b349800aaa4c75193cfba411b Nov 01 15:56:28.677048 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:28.677008 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" event={"ID":"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44","Type":"ContainerStarted","Data":"f6eb252d4fb3131753127d810b34f8017a67e30b349800aaa4c75193cfba411b"} Nov 01 15:56:29.021629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:29.021589 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:56:29.683210 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:29.683179 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" event={"ID":"a0218c49-8d6d-4bb3-8cf7-10634e6c7a44","Type":"ContainerStarted","Data":"67e1b0746c48a43ff49d78c9dfd53bd8457be874c9d5b69e53210e331b705f52"} Nov 01 15:56:29.701989 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:29.701935 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podStartSLOduration=1.5759963799999999 podStartE2EDuration="2.701917282s" totalImagesPullingTime="1.125920902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:29.699825266 +0000 UTC m=+527.402991214" watchObservedRunningTime="2026-11-01 15:56:29.701917282 +0000 UTC m=+527.405083225" Nov 01 15:56:29.940546 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:29.940518 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:30.100319 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:30.100292 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:30.198101 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:30.198030 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:31.274596 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.274562 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:56:31.878419 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.878386 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv"] Nov 01 15:56:31.881800 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.881775 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:31.885131 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.885109 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 01 15:56:31.890693 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.890671 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv"] Nov 01 15:56:31.930649 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.930623 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:31.994324 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.994296 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dgq\" (UniqueName: \"kubernetes.io/projected/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-kube-api-access-72dgq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:31.994505 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.994337 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:31.994505 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.994368 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:31.994505 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.994479 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:31.994505 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.994504 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:31.994661 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:31.994528 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.095558 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.095515 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.095746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.095574 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.095746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.095632 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.095746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.095667 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.095746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.095703 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.095746 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.095728 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dgq\" (UniqueName: \"kubernetes.io/projected/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-kube-api-access-72dgq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.096046 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.095998 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.096103 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.096045 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.096154 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.096135 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.098029 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.098004 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.098192 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.098177 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.104083 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.104063 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dgq\" (UniqueName: \"kubernetes.io/projected/2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c-kube-api-access-72dgq\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-jnxjv\" (UID: \"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.196622 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.196512 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:32.470958 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.470866 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv"] Nov 01 15:56:32.478306 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:56:32.478259 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5d2cae_2c42_4c6f_9cf4_801fc3a67f4c.slice/crio-548541faf16cff773d02a41c542637635a018683347fd1406d5ec10a6409ae07 WatchSource:0}: Error finding container 548541faf16cff773d02a41c542637635a018683347fd1406d5ec10a6409ae07: Status 404 returned error can't find the container with id 548541faf16cff773d02a41c542637635a018683347fd1406d5ec10a6409ae07 Nov 01 15:56:32.696620 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:32.696583 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" event={"ID":"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c","Type":"ContainerStarted","Data":"548541faf16cff773d02a41c542637635a018683347fd1406d5ec10a6409ae07"} Nov 01 15:56:33.702955 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.702922 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" event={"ID":"2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c","Type":"ContainerStarted","Data":"8ea3543738ec1ef9f333a13cc3e93b23d71bb703cc21c5307c235f48f063bad2"} Nov 01 15:56:33.712737 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.712707 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c"] Nov 01 15:56:33.717101 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.717081 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.721730 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.721678 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" podStartSLOduration=2.405065358 podStartE2EDuration="2.721662317s" totalImagesPullingTime="316.596959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:33.719669909 +0000 UTC m=+531.422835846" watchObservedRunningTime="2026-11-01 15:56:33.721662317 +0000 UTC m=+531.424828253" Nov 01 15:56:33.723016 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.722987 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 01 15:56:33.725362 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.725305 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c"] Nov 01 15:56:33.762314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.762284 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:33.810585 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.810554 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.810742 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.810594 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eedaa33d-b796-42e7-8a32-403dc27c0221-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.810742 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.810620 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.810742 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.810722 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjm64\" (UniqueName: \"kubernetes.io/projected/eedaa33d-b796-42e7-8a32-403dc27c0221-kube-api-access-fjm64\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.810929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.810862 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.810984 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.810937 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912012 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.911979 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.912031 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.912047 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eedaa33d-b796-42e7-8a32-403dc27c0221-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.912064 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.912088 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjm64\" (UniqueName: \"kubernetes.io/projected/eedaa33d-b796-42e7-8a32-403dc27c0221-kube-api-access-fjm64\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.912125 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912446 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.912423 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912501 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.912446 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.912501 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.912491 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.914308 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.914285 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eedaa33d-b796-42e7-8a32-403dc27c0221-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.914446 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.914432 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eedaa33d-b796-42e7-8a32-403dc27c0221-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:33.920387 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:33.920363 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjm64\" (UniqueName: \"kubernetes.io/projected/eedaa33d-b796-42e7-8a32-403dc27c0221-kube-api-access-fjm64\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-mrk7c\" (UID: \"eedaa33d-b796-42e7-8a32-403dc27c0221\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:34.034513 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:34.034443 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:34.197388 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:34.197358 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:34.307342 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:34.307319 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c"] Nov 01 15:56:34.309917 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:56:34.309887 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedaa33d_b796_42e7_8a32_403dc27c0221.slice/crio-bc58379fea84a4b71faf7a1a841574c15cd723984507a3f62096beb5b0a473dd WatchSource:0}: Error finding container bc58379fea84a4b71faf7a1a841574c15cd723984507a3f62096beb5b0a473dd: Status 404 returned error can't find the container with id bc58379fea84a4b71faf7a1a841574c15cd723984507a3f62096beb5b0a473dd Nov 01 15:56:34.594713 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:34.594634 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:56:34.710314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:34.710280 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" event={"ID":"eedaa33d-b796-42e7-8a32-403dc27c0221","Type":"ContainerStarted","Data":"cfda1ee2819d38cfe84e365e89fd3c0a31d4c86d8740e873a3a60c1c05217dc0"} Nov 01 15:56:34.710687 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:34.710331 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" event={"ID":"eedaa33d-b796-42e7-8a32-403dc27c0221","Type":"ContainerStarted","Data":"bc58379fea84a4b71faf7a1a841574c15cd723984507a3f62096beb5b0a473dd"} Nov 01 15:56:34.731890 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:34.731838 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" podStartSLOduration=1.4257604640000001 podStartE2EDuration="1.731825922s" totalImagesPullingTime="306.065458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:34.72997282 +0000 UTC m=+532.433138757" watchObservedRunningTime="2026-11-01 15:56:34.731825922 +0000 UTC m=+532.434991858" Nov 01 15:56:35.035303 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:35.035268 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:38.198618 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:38.198581 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:38.211062 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:38.211033 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:38.739592 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:38.739562 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 15:56:42.196640 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:42.196608 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:42.209271 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:42.209245 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:42.757655 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:42.757624 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" Nov 01 15:56:44.034762 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:44.034724 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:44.047327 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:44.047300 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:44.766532 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:44.766502 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" Nov 01 15:56:46.183735 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.183697 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5"] Nov 01 15:56:46.190125 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.190104 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.198808 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.198789 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 15:56:46.199567 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.199541 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tkjjf\"" Nov 01 15:56:46.202272 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.202251 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5"] Nov 01 15:56:46.316358 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316324 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.316358 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316357 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.316572 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316375 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.316572 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316393 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.316572 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316533 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.316572 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316567 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.316709 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316655 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.316709 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316704 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.316821 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.316721 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpqd9\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-kube-api-access-vpqd9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418047 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418015 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418185 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418069 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418185 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqd9\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-kube-api-access-vpqd9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418272 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418255 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418304 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418287 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418345 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418313 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418452 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418432 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418536 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418452 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418536 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418516 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418629 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418547 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418695 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418672 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418889 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418866 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.418962 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418921 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.419011 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.418978 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.420572 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.420554 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.420734 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.420717 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.425399 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.425377 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.425544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.425432 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpqd9\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-kube-api-access-vpqd9\") pod \"maas-default-gateway-openshift-default-7f5667f86c-kkcz5\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.505274 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.505176 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:46.788297 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:46.788271 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5"] Nov 01 15:56:46.790861 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:56:46.790830 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37db937e_ba95_48cd_ac9d_d93d4c28d0cf.slice/crio-80ed4feb896a3e0000b32938082761fd4174419031ff13d0890615d8dd2b1b16 WatchSource:0}: Error finding container 80ed4feb896a3e0000b32938082761fd4174419031ff13d0890615d8dd2b1b16: Status 404 returned error can't find the container with id 80ed4feb896a3e0000b32938082761fd4174419031ff13d0890615d8dd2b1b16 Nov 01 15:56:47.770421 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:47.770384 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" event={"ID":"37db937e-ba95-48cd-ac9d-d93d4c28d0cf","Type":"ContainerStarted","Data":"80ed4feb896a3e0000b32938082761fd4174419031ff13d0890615d8dd2b1b16"} Nov 01 15:56:49.431821 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:49.431733 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:56:50.039881 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:50.039851 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:50.039998 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:50.039917 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:50.039998 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:50.039945 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:56:50.794248 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:50.794211 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" event={"ID":"37db937e-ba95-48cd-ac9d-d93d4c28d0cf","Type":"ContainerStarted","Data":"e4e5b265dea7a8c03af10ef341b69ce376670a0ee8108c75e4b3631c3d682e1b"} Nov 01 15:56:50.813125 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:50.813063 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" podStartSLOduration=1.5669680540000002 podStartE2EDuration="4.813050931s" totalImagesPullingTime="3.246082877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:56:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:56:50.813031329 +0000 UTC m=+548.516197266" watchObservedRunningTime="2026-11-01 15:56:50.813050931 +0000 UTC m=+548.516216866" Nov 01 15:56:51.505719 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:51.505683 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:51.507252 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:51.507231 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kkcz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:56:51.507322 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:51.507305 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 15:56:52.506388 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:52.506351 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kkcz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:56:52.506900 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:52.506412 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 15:56:53.506521 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:53.506487 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kkcz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:56:53.506946 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:53.506541 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 15:56:54.376329 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:56:54.376250 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:56:54.506177 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:54.506141 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kkcz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:56:54.506335 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:54.506199 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 15:56:55.506103 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:55.506065 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kkcz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:56:55.506502 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:55.506124 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 15:56:56.506338 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:56.506299 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-kkcz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 01 15:56:56.506747 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:56.506358 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 01 15:56:58.199552 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:58.199521 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:58.200048 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:58.199852 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:56:58.200645 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:56:58.200625 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 15:57:01.227121 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.227078 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb"] Nov 01 15:57:01.230786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.230739 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.246382 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.246359 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb"] Nov 01 15:57:01.274168 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274139 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.274343 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274173 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.274343 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274199 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.274454 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274360 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf4jf\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-kube-api-access-lf4jf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.274505 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274450 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.274542 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274504 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.274542 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274522 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.274634 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274553 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.274697 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.274684 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.375612 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375578 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.375856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375621 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.375856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375641 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.375856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375665 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.375856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375693 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf4jf\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-kube-api-access-lf4jf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.375856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375729 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.375856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375787 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.376180 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375856 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.376180 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.375923 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.376180 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.376083 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.376304 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.376208 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.376304 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.376234 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.376304 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.376282 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.376570 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.376549 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.377904 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.377883 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.378200 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.378179 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.384619 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.384597 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf4jf\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-kube-api-access-lf4jf\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.384694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.384613 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-klsmb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.545436 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.545325 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:01.821477 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.821451 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb"] Nov 01 15:57:01.823699 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:57:01.823667 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af93991_fb33_4054_82cd_5e76ebfa4cbb.slice/crio-08c3885808177258ccf876760931f67c018ea2d65d903f3c6c7bc1760604b26c WatchSource:0}: Error finding container 08c3885808177258ccf876760931f67c018ea2d65d903f3c6c7bc1760604b26c: Status 404 returned error can't find the container with id 08c3885808177258ccf876760931f67c018ea2d65d903f3c6c7bc1760604b26c Nov 01 15:57:01.827018 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.826983 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.827097 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.827068 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.827141 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.827107 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:01.847566 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:01.847531 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" event={"ID":"1af93991-fb33-4054-82cd-5e76ebfa4cbb","Type":"ContainerStarted","Data":"08c3885808177258ccf876760931f67c018ea2d65d903f3c6c7bc1760604b26c"} Nov 01 15:57:02.854092 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:02.854055 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" event={"ID":"1af93991-fb33-4054-82cd-5e76ebfa4cbb","Type":"ContainerStarted","Data":"1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228"} Nov 01 15:57:02.882775 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:02.876546 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" podStartSLOduration=1.876528151 podStartE2EDuration="1.876528151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:02.874222491 +0000 UTC m=+560.577388428" watchObservedRunningTime="2026-11-01 15:57:02.876528151 +0000 UTC m=+560.579694089" Nov 01 15:57:03.546316 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:03.546277 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:03.547398 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:03.547372 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-klsmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:57:03.547493 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:03.547432 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 15:57:04.546318 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:04.546280 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-klsmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:57:04.546718 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:04.546361 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 15:57:05.546077 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:05.546046 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-klsmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:57:05.546246 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:05.546106 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 15:57:06.546202 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:06.546169 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-klsmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:57:06.546582 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:06.546218 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 15:57:07.546016 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:07.545982 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-klsmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:57:07.546225 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:07.546039 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 15:57:08.546093 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:08.546052 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-klsmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 01 15:57:08.546266 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:08.546121 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 01 15:57:10.037881 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:10.037848 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:10.038298 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:10.038234 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:10.039030 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:10.039014 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 15:57:12.763730 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:12.763698 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:57:15.612971 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.612937 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6674db54f7-rpgrj"] Nov 01 15:57:15.617697 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.617674 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:15.623340 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.623316 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 01 15:57:15.623963 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.623472 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-lxbnx\"" Nov 01 15:57:15.626041 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.626019 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6674db54f7-rpgrj"] Nov 01 15:57:15.626124 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.626062 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:15.713533 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.713502 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 01 15:57:15.717987 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.717959 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48h94\" (UniqueName: \"kubernetes.io/projected/04460888-e8b0-4f27-961f-94df59fcb299-kube-api-access-48h94\") pod \"ai-gateway-controller-6674db54f7-rpgrj\" (UID: \"04460888-e8b0-4f27-961f-94df59fcb299\") " pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:15.718123 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.718026 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04460888-e8b0-4f27-961f-94df59fcb299-metrics-certs\") pod \"ai-gateway-controller-6674db54f7-rpgrj\" (UID: \"04460888-e8b0-4f27-961f-94df59fcb299\") " pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:15.723616 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.723592 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:15.819541 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.819509 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48h94\" (UniqueName: \"kubernetes.io/projected/04460888-e8b0-4f27-961f-94df59fcb299-kube-api-access-48h94\") pod \"ai-gateway-controller-6674db54f7-rpgrj\" (UID: \"04460888-e8b0-4f27-961f-94df59fcb299\") " pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:15.819698 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.819556 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04460888-e8b0-4f27-961f-94df59fcb299-metrics-certs\") pod \"ai-gateway-controller-6674db54f7-rpgrj\" (UID: \"04460888-e8b0-4f27-961f-94df59fcb299\") " pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:15.821972 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.821943 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04460888-e8b0-4f27-961f-94df59fcb299-metrics-certs\") pod \"ai-gateway-controller-6674db54f7-rpgrj\" (UID: \"04460888-e8b0-4f27-961f-94df59fcb299\") " pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:15.830472 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.830452 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48h94\" (UniqueName: \"kubernetes.io/projected/04460888-e8b0-4f27-961f-94df59fcb299-kube-api-access-48h94\") pod \"ai-gateway-controller-6674db54f7-rpgrj\" (UID: \"04460888-e8b0-4f27-961f-94df59fcb299\") " pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:15.932354 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:15.932284 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:16.209456 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.209414 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6674db54f7-rpgrj"] Nov 01 15:57:16.211716 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:57:16.211686 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04460888_e8b0_4f27_961f_94df59fcb299.slice/crio-5f3cc4887a99dcd58a169dd0c29c20bd0a0c44cda257a09a80320e4703994e51 WatchSource:0}: Error finding container 5f3cc4887a99dcd58a169dd0c29c20bd0a0c44cda257a09a80320e4703994e51: Status 404 returned error can't find the container with id 5f3cc4887a99dcd58a169dd0c29c20bd0a0c44cda257a09a80320e4703994e51 Nov 01 15:57:16.236179 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.236150 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm"] Nov 01 15:57:16.240651 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.240632 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.254065 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.254041 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm"] Nov 01 15:57:16.324034 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324000 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.324034 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324030 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.324207 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324127 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.324207 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324185 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.324270 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324209 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.324270 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324231 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.324270 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324247 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.324389 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324376 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cbf3fc-1566-4516-9a74-c7684e42fe01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.324438 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.324424 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbwcr\" (UniqueName: \"kubernetes.io/projected/73cbf3fc-1566-4516-9a74-c7684e42fe01-kube-api-access-kbwcr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.425670 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.425633 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.425670 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.425667 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.425926 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.425690 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cbf3fc-1566-4516-9a74-c7684e42fe01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.425926 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.425745 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbwcr\" (UniqueName: \"kubernetes.io/projected/73cbf3fc-1566-4516-9a74-c7684e42fe01-kube-api-access-kbwcr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.425926 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.425840 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.425926 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.425866 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.426127 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.425929 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.426127 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.425982 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.426127 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.426012 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.426127 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.426112 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.426305 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.426290 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.426367 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.426344 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.426485 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.426470 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.426672 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.426656 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73cbf3fc-1566-4516-9a74-c7684e42fe01-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.428273 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.428249 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.428366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.428274 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.438394 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.438368 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbwcr\" (UniqueName: \"kubernetes.io/projected/73cbf3fc-1566-4516-9a74-c7684e42fe01-kube-api-access-kbwcr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.439071 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.439054 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73cbf3fc-1566-4516-9a74-c7684e42fe01-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-v67pm\" (UID: \"73cbf3fc-1566-4516-9a74-c7684e42fe01\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.554575 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.554485 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:16.830162 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.830131 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm"] Nov 01 15:57:16.832647 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:57:16.832621 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cbf3fc_1566_4516_9a74_c7684e42fe01.slice/crio-1e202249a4e91089a3502b7ca23feede8726b8b55aa4a3fa7df427fcd99c49db WatchSource:0}: Error finding container 1e202249a4e91089a3502b7ca23feede8726b8b55aa4a3fa7df427fcd99c49db: Status 404 returned error can't find the container with id 1e202249a4e91089a3502b7ca23feede8726b8b55aa4a3fa7df427fcd99c49db Nov 01 15:57:16.835207 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.835179 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:16.835275 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.835239 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:16.835275 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.835271 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:57:16.944170 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.944134 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" event={"ID":"73cbf3fc-1566-4516-9a74-c7684e42fe01","Type":"ContainerStarted","Data":"fd74195ee7c1a7eea5134a995c83b13ed5cd1b51ff9570255b61df86a1e883c0"} Nov 01 15:57:16.944363 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.944194 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" event={"ID":"73cbf3fc-1566-4516-9a74-c7684e42fe01","Type":"ContainerStarted","Data":"1e202249a4e91089a3502b7ca23feede8726b8b55aa4a3fa7df427fcd99c49db"} Nov 01 15:57:16.945526 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.945499 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" event={"ID":"04460888-e8b0-4f27-961f-94df59fcb299","Type":"ContainerStarted","Data":"5f3cc4887a99dcd58a169dd0c29c20bd0a0c44cda257a09a80320e4703994e51"} Nov 01 15:57:16.967050 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:16.966994 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podStartSLOduration=0.96697547 podStartE2EDuration="966.97547ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:16.964603382 +0000 UTC m=+574.667769320" watchObservedRunningTime="2026-11-01 15:57:16.96697547 +0000 UTC m=+574.670141408" Nov 01 15:57:17.555334 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:17.555286 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:17.556601 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:17.556560 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 15:57:17.556733 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:17.556626 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 15:57:18.555919 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:18.555438 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 15:57:18.555919 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:18.555517 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 15:57:19.554990 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:19.554954 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 15:57:19.555196 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:19.555015 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 15:57:20.555210 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:20.555179 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 15:57:20.555642 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:20.555234 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 15:57:21.555855 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:21.555816 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 15:57:21.556232 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:21.555871 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 15:57:21.997321 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:21.997284 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" event={"ID":"04460888-e8b0-4f27-961f-94df59fcb299","Type":"ContainerStarted","Data":"7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b"} Nov 01 15:57:21.997492 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:21.997362 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:22.013970 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.013924 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" podStartSLOduration=2.322407532 podStartE2EDuration="7.013911922s" totalImagesPullingTime="4.69150439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:22.013071789 +0000 UTC m=+579.716237730" watchObservedRunningTime="2026-11-01 15:57:22.013911922 +0000 UTC m=+579.717077858" Nov 01 15:57:22.555318 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.555280 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 15:57:22.555522 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.555342 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 15:57:22.716024 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.715991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:22.797458 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.797418 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-44k4f"] Nov 01 15:57:22.801186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.801164 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:22.809510 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.809486 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 01 15:57:22.810595 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.809993 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-mkx9l\"" Nov 01 15:57:22.810595 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.810038 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-44k4f"] Nov 01 15:57:22.811194 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.811174 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:22.923263 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.923215 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5127857-fa25-4c87-8958-69cfb960d73f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-44k4f\" (UID: \"e5127857-fa25-4c87-8958-69cfb960d73f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:22.923468 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:22.923361 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt9bq\" (UniqueName: \"kubernetes.io/projected/e5127857-fa25-4c87-8958-69cfb960d73f-kube-api-access-bt9bq\") pod \"payload-pre-processing-7b98bb678c-44k4f\" (UID: \"e5127857-fa25-4c87-8958-69cfb960d73f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:23.024135 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:23.024097 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt9bq\" (UniqueName: \"kubernetes.io/projected/e5127857-fa25-4c87-8958-69cfb960d73f-kube-api-access-bt9bq\") pod \"payload-pre-processing-7b98bb678c-44k4f\" (UID: \"e5127857-fa25-4c87-8958-69cfb960d73f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:23.024335 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:23.024223 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5127857-fa25-4c87-8958-69cfb960d73f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-44k4f\" (UID: \"e5127857-fa25-4c87-8958-69cfb960d73f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:23.024936 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:23.024915 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5127857-fa25-4c87-8958-69cfb960d73f-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-44k4f\" (UID: \"e5127857-fa25-4c87-8958-69cfb960d73f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:23.032535 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:23.032510 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt9bq\" (UniqueName: \"kubernetes.io/projected/e5127857-fa25-4c87-8958-69cfb960d73f-kube-api-access-bt9bq\") pod \"payload-pre-processing-7b98bb678c-44k4f\" (UID: \"e5127857-fa25-4c87-8958-69cfb960d73f\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:23.118226 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:23.118187 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:23.397584 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:23.397549 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-44k4f"] Nov 01 15:57:23.399972 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:57:23.399948 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5127857_fa25_4c87_8958_69cfb960d73f.slice/crio-2be19d2ce5935a96492e28594c2606eafe7f154119bb20cb7f637bdc14d62e53 WatchSource:0}: Error finding container 2be19d2ce5935a96492e28594c2606eafe7f154119bb20cb7f637bdc14d62e53: Status 404 returned error can't find the container with id 2be19d2ce5935a96492e28594c2606eafe7f154119bb20cb7f637bdc14d62e53 Nov 01 15:57:23.555408 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:23.555372 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body= Nov 01 15:57:23.555603 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:23.555423 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" Nov 01 15:57:24.008696 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:24.008658 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" event={"ID":"e5127857-fa25-4c87-8958-69cfb960d73f","Type":"ContainerStarted","Data":"2be19d2ce5935a96492e28594c2606eafe7f154119bb20cb7f637bdc14d62e53"} Nov 01 15:57:24.372063 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:57:24.371947 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:57:25.556844 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:25.556382 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:57:25.556844 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:25.556441 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:57:26.022059 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:26.022018 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" event={"ID":"e5127857-fa25-4c87-8958-69cfb960d73f","Type":"ContainerStarted","Data":"9fb844c7876f9501b51a7a7716e4d1c5a60f06e1152567f56e4419914d5f5133"} Nov 01 15:57:26.022241 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:26.022171 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:26.040057 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:26.039993 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" podStartSLOduration=1.805327795 podStartE2EDuration="4.039972899s" totalImagesPullingTime="2.234645104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:26.037362447 +0000 UTC m=+583.740528382" watchObservedRunningTime="2026-11-01 15:57:26.039972899 +0000 UTC m=+583.743138836" Nov 01 15:57:26.557043 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:26.557003 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:57:26.557458 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:26.557060 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:57:27.413504 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:27.413473 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:27.413874 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:27.413854 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:27.414524 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:27.414508 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" Nov 01 15:57:29.973608 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:29.973553 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:33.004924 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.004891 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:33.263710 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.263634 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7847bd554f-lqsgd"] Nov 01 15:57:33.267338 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.267321 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:33.278290 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.278262 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7847bd554f-lqsgd"] Nov 01 15:57:33.325267 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.325229 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg2xw\" (UniqueName: \"kubernetes.io/projected/31af0e80-54db-44bf-8849-289f596b400e-kube-api-access-jg2xw\") pod \"ai-gateway-controller-7847bd554f-lqsgd\" (UID: \"31af0e80-54db-44bf-8849-289f596b400e\") " pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:33.325439 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.325298 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31af0e80-54db-44bf-8849-289f596b400e-metrics-certs\") pod \"ai-gateway-controller-7847bd554f-lqsgd\" (UID: \"31af0e80-54db-44bf-8849-289f596b400e\") " pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:33.426317 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.426281 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg2xw\" (UniqueName: \"kubernetes.io/projected/31af0e80-54db-44bf-8849-289f596b400e-kube-api-access-jg2xw\") pod \"ai-gateway-controller-7847bd554f-lqsgd\" (UID: \"31af0e80-54db-44bf-8849-289f596b400e\") " pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:33.426477 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.426335 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31af0e80-54db-44bf-8849-289f596b400e-metrics-certs\") pod \"ai-gateway-controller-7847bd554f-lqsgd\" (UID: \"31af0e80-54db-44bf-8849-289f596b400e\") " pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:33.428793 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.428738 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31af0e80-54db-44bf-8849-289f596b400e-metrics-certs\") pod \"ai-gateway-controller-7847bd554f-lqsgd\" (UID: \"31af0e80-54db-44bf-8849-289f596b400e\") " pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:33.433999 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.433977 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg2xw\" (UniqueName: \"kubernetes.io/projected/31af0e80-54db-44bf-8849-289f596b400e-kube-api-access-jg2xw\") pod \"ai-gateway-controller-7847bd554f-lqsgd\" (UID: \"31af0e80-54db-44bf-8849-289f596b400e\") " pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:33.586596 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.586509 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:33.861097 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:33.861071 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7847bd554f-lqsgd"] Nov 01 15:57:33.863321 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:57:33.863296 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31af0e80_54db_44bf_8849_289f596b400e.slice/crio-c2f2d28a146990775f93f0d911e443159d60ff86c8f120bffe13c15cd43ed254 WatchSource:0}: Error finding container c2f2d28a146990775f93f0d911e443159d60ff86c8f120bffe13c15cd43ed254: Status 404 returned error can't find the container with id c2f2d28a146990775f93f0d911e443159d60ff86c8f120bffe13c15cd43ed254 Nov 01 15:57:34.063058 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:34.063023 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" event={"ID":"31af0e80-54db-44bf-8849-289f596b400e","Type":"ContainerStarted","Data":"c2f2d28a146990775f93f0d911e443159d60ff86c8f120bffe13c15cd43ed254"} Nov 01 15:57:35.068924 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:35.068887 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" event={"ID":"31af0e80-54db-44bf-8849-289f596b400e","Type":"ContainerStarted","Data":"7e3deff4ea5b2055cf805367d09595193114bf56aee5dc351e993ef4c85d9d00"} Nov 01 15:57:35.069450 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:35.069017 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:35.084610 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:35.084563 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" podStartSLOduration=1.641406967 podStartE2EDuration="2.084551879s" totalImagesPullingTime="443.144912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:57:35.083190354 +0000 UTC m=+592.786356291" watchObservedRunningTime="2026-11-01 15:57:35.084551879 +0000 UTC m=+592.787717814" Nov 01 15:57:37.030590 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:37.030559 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-44k4f" Nov 01 15:57:42.500065 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:42.500039 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 15:57:42.504543 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:42.504519 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 15:57:42.525031 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:42.525004 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:57:42.528102 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:42.528081 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 15:57:46.076692 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.076662 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7847bd554f-lqsgd" Nov 01 15:57:46.119973 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.119938 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6674db54f7-rpgrj"] Nov 01 15:57:46.120247 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.120175 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" podUID="04460888-e8b0-4f27-961f-94df59fcb299" containerName="manager" containerID="cri-o://7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b" gracePeriod=30 Nov 01 15:57:46.370876 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.370849 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:46.457817 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.457783 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04460888-e8b0-4f27-961f-94df59fcb299-kube-api-access-48h94\" (UniqueName: \"kubernetes.io/projected/04460888-e8b0-4f27-961f-94df59fcb299-kube-api-access-48h94\") pod \"04460888-e8b0-4f27-961f-94df59fcb299\" (UID: \"04460888-e8b0-4f27-961f-94df59fcb299\") " Nov 01 15:57:46.457963 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.457844 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04460888-e8b0-4f27-961f-94df59fcb299-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04460888-e8b0-4f27-961f-94df59fcb299-metrics-certs\") pod \"04460888-e8b0-4f27-961f-94df59fcb299\" (UID: \"04460888-e8b0-4f27-961f-94df59fcb299\") " Nov 01 15:57:46.459891 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.459862 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04460888-e8b0-4f27-961f-94df59fcb299-metrics-certs" pod "04460888-e8b0-4f27-961f-94df59fcb299" (UID: "04460888-e8b0-4f27-961f-94df59fcb299"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 15:57:46.460008 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.459970 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04460888-e8b0-4f27-961f-94df59fcb299-kube-api-access-48h94" pod "04460888-e8b0-4f27-961f-94df59fcb299" (UID: "04460888-e8b0-4f27-961f-94df59fcb299"). InnerVolumeSpecName "kube-api-access-48h94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 15:57:46.559554 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.559510 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04460888-e8b0-4f27-961f-94df59fcb299-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:57:46.559554 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:46.559550 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48h94\" (UniqueName: \"kubernetes.io/projected/04460888-e8b0-4f27-961f-94df59fcb299-kube-api-access-48h94\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 15:57:47.130086 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.130044 4573 generic.go:356] "Generic (PLEG): container finished" podID="04460888-e8b0-4f27-961f-94df59fcb299" containerID="7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b" exitCode=0 Nov 01 15:57:47.130086 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.130086 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" Nov 01 15:57:47.130626 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.130121 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" event={"ID":"04460888-e8b0-4f27-961f-94df59fcb299","Type":"ContainerDied","Data":"7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b"} Nov 01 15:57:47.130626 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.130172 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6674db54f7-rpgrj" event={"ID":"04460888-e8b0-4f27-961f-94df59fcb299","Type":"ContainerDied","Data":"5f3cc4887a99dcd58a169dd0c29c20bd0a0c44cda257a09a80320e4703994e51"} Nov 01 15:57:47.130626 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.130183 4573 scope.go:122] "RemoveContainer" containerID="7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b" Nov 01 15:57:47.139402 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.139384 4573 scope.go:122] "RemoveContainer" containerID="7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b" Nov 01 15:57:47.139656 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:57:47.139637 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b\": container with ID starting with 7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b not found: ID does not exist" containerID="7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b" Nov 01 15:57:47.139694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.139665 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b"} err="failed to get container status \"7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b\": rpc error: code = NotFound desc = could not find container \"7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b\": container with ID starting with 7f199a7b4459f0dcd8ed60fe031eb38070260054aac9679fd285c9a50f8d3d1b not found: ID does not exist" Nov 01 15:57:47.151656 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.151633 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-6674db54f7-rpgrj"] Nov 01 15:57:47.156257 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.156236 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-6674db54f7-rpgrj"] Nov 01 15:57:47.703556 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:47.703522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:48.202322 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:48.202287 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:48.456225 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:48.456142 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04460888-e8b0-4f27-961f-94df59fcb299" path="/var/lib/kubelet/pods/04460888-e8b0-4f27-961f-94df59fcb299/volumes" Nov 01 15:57:49.767992 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:49.767954 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:49.868440 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:49.868407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:51.241369 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:51.241338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:51.343482 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:51.343454 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:53.485847 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:57:53.485815 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:57:54.362608 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:57:54.362575 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:58:12.656343 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:12.656295 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:12.758457 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:12.758418 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:13.074151 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.074075 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws"] Nov 01 15:58:13.074554 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.074540 4573 cpu_manager.go:414] "containerMap: removing container" podUID="04460888-e8b0-4f27-961f-94df59fcb299" containerName="manager" Nov 01 15:58:13.074598 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.074555 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04460888-e8b0-4f27-961f-94df59fcb299" containerName="manager" Nov 01 15:58:13.074639 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.074629 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="04460888-e8b0-4f27-961f-94df59fcb299" containerName="manager" Nov 01 15:58:13.081142 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.081123 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.109545 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.109509 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.109684 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.109565 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.109770 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.109704 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.109846 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.109774 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.109904 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.109852 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.109966 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.109946 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.110010 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.109989 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-546gl\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-kube-api-access-546gl\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.110057 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.110014 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.110108 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.110093 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59186df5-07c5-4dbc-b45d-d474e1514fe6-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.112346 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.112329 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-43fac8-openshift-default-dockercfg-4dq2m\"" Nov 01 15:58:13.117267 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.117242 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws"] Nov 01 15:58:13.210926 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.210893 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211085 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.210941 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211085 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.210968 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211085 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211037 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211085 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211077 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-546gl\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-kube-api-access-546gl\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211275 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211113 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211275 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211137 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59186df5-07c5-4dbc-b45d-d474e1514fe6-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211275 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211163 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211275 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211199 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211339 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211400 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211453 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211631 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211600 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.211953 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.211934 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59186df5-07c5-4dbc-b45d-d474e1514fe6-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.213551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.213523 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.213900 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.213876 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.244570 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.244536 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.244687 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.244657 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-546gl\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-kube-api-access-546gl\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.351165 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.351079 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:13.394140 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.394109 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:13.416854 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.416172 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f"] Nov 01 15:58:13.421290 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.420841 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.426461 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.425296 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-4a083d-openshift-default-dockercfg-qxpcr\"" Nov 01 15:58:13.446797 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.445982 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f"] Nov 01 15:58:13.485026 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.484990 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:13.515168 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515138 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.515314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515175 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.515314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515200 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747c189-9aad-474e-b650-79061531f01d-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.515314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515217 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.515314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515244 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.515314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515267 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.515314 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515288 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747c189-9aad-474e-b650-79061531f01d-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.515511 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515341 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.515511 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.515362 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zpfq\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-kube-api-access-6zpfq\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616147 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616116 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616334 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616156 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zpfq\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-kube-api-access-6zpfq\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616334 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616202 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616334 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616221 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616334 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616255 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747c189-9aad-474e-b650-79061531f01d-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616334 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616284 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616334 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616330 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616641 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616361 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616641 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616384 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747c189-9aad-474e-b650-79061531f01d-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616849 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616660 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616849 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616732 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.616951 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.616858 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.617038 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.617019 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747c189-9aad-474e-b650-79061531f01d-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.617214 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.617191 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.618772 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.618733 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.619122 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.619104 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747c189-9aad-474e-b650-79061531f01d-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.642247 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.642216 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.642457 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.642427 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zpfq\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-kube-api-access-6zpfq\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.710477 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.710445 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws"] Nov 01 15:58:13.714335 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.714279 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:13.714980 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:13.714959 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59186df5_07c5_4dbc_b45d_d474e1514fe6.slice/crio-736ded45db8fc1639cb66a356366b767c60b4ff9b901a95dce9e97a034ff0dce WatchSource:0}: Error finding container 736ded45db8fc1639cb66a356366b767c60b4ff9b901a95dce9e97a034ff0dce: Status 404 returned error can't find the container with id 736ded45db8fc1639cb66a356366b767c60b4ff9b901a95dce9e97a034ff0dce Nov 01 15:58:13.717188 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.717158 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:13.717280 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.717234 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:13.717346 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.717283 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:13.749277 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.749257 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:13.823882 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.823778 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:13.933347 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:13.933253 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:14.070016 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:14.069898 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5747c189_9aad_474e_b650_79061531f01d.slice/crio-78c7b2091756a764e493ec547426e823c6647d99d705e10923ddece3401a18d9 WatchSource:0}: Error finding container 78c7b2091756a764e493ec547426e823c6647d99d705e10923ddece3401a18d9: Status 404 returned error can't find the container with id 78c7b2091756a764e493ec547426e823c6647d99d705e10923ddece3401a18d9 Nov 01 15:58:14.072513 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.072483 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:14.073321 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.073292 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f"] Nov 01 15:58:14.073429 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.073317 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.073429 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.073372 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.073429 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.073417 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:14.163452 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.163423 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:14.259071 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.258977 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" event={"ID":"59186df5-07c5-4dbc-b45d-d474e1514fe6","Type":"ContainerStarted","Data":"36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a"} Nov 01 15:58:14.259071 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.259048 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" event={"ID":"59186df5-07c5-4dbc-b45d-d474e1514fe6","Type":"ContainerStarted","Data":"736ded45db8fc1639cb66a356366b767c60b4ff9b901a95dce9e97a034ff0dce"} Nov 01 15:58:14.260663 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.260635 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" event={"ID":"5747c189-9aad-474e-b650-79061531f01d","Type":"ContainerStarted","Data":"2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c"} Nov 01 15:58:14.260821 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.260675 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" event={"ID":"5747c189-9aad-474e-b650-79061531f01d","Type":"ContainerStarted","Data":"78c7b2091756a764e493ec547426e823c6647d99d705e10923ddece3401a18d9"} Nov 01 15:58:14.278136 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.278085 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podStartSLOduration=1.278073702 podStartE2EDuration="1.278073702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:14.277966102 +0000 UTC m=+631.981132050" watchObservedRunningTime="2026-11-01 15:58:14.278073702 +0000 UTC m=+631.981239637" Nov 01 15:58:14.298150 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.298107 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podStartSLOduration=1.2980935599999999 podStartE2EDuration="1.29809356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:14.295821388 +0000 UTC m=+631.998987324" watchObservedRunningTime="2026-11-01 15:58:14.29809356 +0000 UTC m=+632.001259497" Nov 01 15:58:14.395082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.395047 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:14.400615 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.400581 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:14.750161 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:14.750133 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:15.266179 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:15.266141 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:15.266982 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:15.266964 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:15.756167 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:15.756135 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:16.271001 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:16.270972 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:16.271984 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:16.271959 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:16.548784 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:16.548673 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:16.661969 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:16.661935 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:16.759326 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:16.759294 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:16.864856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:16.864826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:16.966615 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:16.966578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:17.752539 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.752503 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:17.846816 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.846778 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:17.881515 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.881481 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs"] Nov 01 15:58:17.886093 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.886074 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:17.892308 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.892283 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs"] Nov 01 15:58:17.955870 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.955841 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-43fac8-dockercfg-t8qgw\"" Nov 01 15:58:17.956101 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.955835 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-43fac8\"" Nov 01 15:58:17.964633 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.964599 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc58v\" (UniqueName: \"kubernetes.io/projected/31dd6eb6-7339-4276-8332-e89ec52d1d5e-kube-api-access-jc58v\") pod \"payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs\" (UID: \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:17.965041 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:17.964726 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd6eb6-7339-4276-8332-e89ec52d1d5e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs\" (UID: \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:18.066071 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.065987 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc58v\" (UniqueName: \"kubernetes.io/projected/31dd6eb6-7339-4276-8332-e89ec52d1d5e-kube-api-access-jc58v\") pod \"payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs\" (UID: \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:18.066217 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.066101 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd6eb6-7339-4276-8332-e89ec52d1d5e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs\" (UID: \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:18.066910 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.066883 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd6eb6-7339-4276-8332-e89ec52d1d5e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs\" (UID: \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:18.074263 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.074228 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc58v\" (UniqueName: \"kubernetes.io/projected/31dd6eb6-7339-4276-8332-e89ec52d1d5e-kube-api-access-jc58v\") pod \"payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs\" (UID: \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:18.201992 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.201960 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:18.446939 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.446905 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:18.481610 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.481582 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs"] Nov 01 15:58:18.482611 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:18.482590 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dd6eb6_7339_4276_8332_e89ec52d1d5e.slice/crio-d74029debd527bf23bf71a093fec333eb1dc8939c3d557242b4a6c3faed8ac93 WatchSource:0}: Error finding container d74029debd527bf23bf71a093fec333eb1dc8939c3d557242b4a6c3faed8ac93: Status 404 returned error can't find the container with id d74029debd527bf23bf71a093fec333eb1dc8939c3d557242b4a6c3faed8ac93 Nov 01 15:58:18.516812 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.516779 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm"] Nov 01 15:58:18.521616 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.521597 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:18.531783 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.531731 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-4a083d\"" Nov 01 15:58:18.532425 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.532348 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm"] Nov 01 15:58:18.563166 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.563123 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:18.575117 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.575094 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-4a083d-dockercfg-lpws5\"" Nov 01 15:58:18.583962 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.583939 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae37fe0-f2ca-4b70-9ac8-27843820b864-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm\" (UID: \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:18.584253 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.584219 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvsc\" (UniqueName: \"kubernetes.io/projected/5ae37fe0-f2ca-4b70-9ac8-27843820b864-kube-api-access-5zvsc\") pod \"payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm\" (UID: \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:18.685454 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.685416 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae37fe0-f2ca-4b70-9ac8-27843820b864-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm\" (UID: \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:18.685642 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.685540 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvsc\" (UniqueName: \"kubernetes.io/projected/5ae37fe0-f2ca-4b70-9ac8-27843820b864-kube-api-access-5zvsc\") pod \"payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm\" (UID: \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:18.686280 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.686251 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae37fe0-f2ca-4b70-9ac8-27843820b864-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm\" (UID: \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:18.686694 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.686674 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:18.693287 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.693265 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvsc\" (UniqueName: \"kubernetes.io/projected/5ae37fe0-f2ca-4b70-9ac8-27843820b864-kube-api-access-5zvsc\") pod \"payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm\" (UID: \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:18.786981 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.786901 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:18.850895 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.850862 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:18.902441 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.902405 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:18.981676 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.981644 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq"] Nov 01 15:58:18.986263 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.986247 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:18.993847 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.993542 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-164db6\"" Nov 01 15:58:18.998919 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:18.998894 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq"] Nov 01 15:58:19.081443 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.081356 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-164db6-dockercfg-tdx2p\"" Nov 01 15:58:19.088994 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.088957 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e04b490-5013-4ff0-bd43-c529b49a559f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq\" (UID: \"2e04b490-5013-4ff0-bd43-c529b49a559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:19.089174 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.089146 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js4f4\" (UniqueName: \"kubernetes.io/projected/2e04b490-5013-4ff0-bd43-c529b49a559f-kube-api-access-js4f4\") pod \"payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq\" (UID: \"2e04b490-5013-4ff0-bd43-c529b49a559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:19.115170 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.115139 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:19.136010 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.135328 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm"] Nov 01 15:58:19.137090 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:19.137039 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae37fe0_f2ca_4b70_9ac8_27843820b864.slice/crio-c00b798c26242a351e3d1586d3d504dca7fb46170de825a8e3c9ba070ccebae6 WatchSource:0}: Error finding container c00b798c26242a351e3d1586d3d504dca7fb46170de825a8e3c9ba070ccebae6: Status 404 returned error can't find the container with id c00b798c26242a351e3d1586d3d504dca7fb46170de825a8e3c9ba070ccebae6 Nov 01 15:58:19.189901 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.189872 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e04b490-5013-4ff0-bd43-c529b49a559f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq\" (UID: \"2e04b490-5013-4ff0-bd43-c529b49a559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:19.190031 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.189992 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js4f4\" (UniqueName: \"kubernetes.io/projected/2e04b490-5013-4ff0-bd43-c529b49a559f-kube-api-access-js4f4\") pod \"payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq\" (UID: \"2e04b490-5013-4ff0-bd43-c529b49a559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:19.190592 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.190573 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e04b490-5013-4ff0-bd43-c529b49a559f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq\" (UID: \"2e04b490-5013-4ff0-bd43-c529b49a559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:19.199343 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.199321 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js4f4\" (UniqueName: \"kubernetes.io/projected/2e04b490-5013-4ff0-bd43-c529b49a559f-kube-api-access-js4f4\") pod \"payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq\" (UID: \"2e04b490-5013-4ff0-bd43-c529b49a559f\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:19.206245 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.206218 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:19.300960 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.300922 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" event={"ID":"5ae37fe0-f2ca-4b70-9ac8-27843820b864","Type":"ContainerStarted","Data":"3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211"} Nov 01 15:58:19.301154 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.301053 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" event={"ID":"5ae37fe0-f2ca-4b70-9ac8-27843820b864","Type":"ContainerStarted","Data":"c00b798c26242a351e3d1586d3d504dca7fb46170de825a8e3c9ba070ccebae6"} Nov 01 15:58:19.301154 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.301089 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:19.302804 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.302745 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" event={"ID":"31dd6eb6-7339-4276-8332-e89ec52d1d5e","Type":"ContainerStarted","Data":"2993fc65690e1d5baa0516df25ea6ea4b50c2ded10775ddeb41e3be76fdbb468"} Nov 01 15:58:19.302916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.302830 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" event={"ID":"31dd6eb6-7339-4276-8332-e89ec52d1d5e","Type":"ContainerStarted","Data":"d74029debd527bf23bf71a093fec333eb1dc8939c3d557242b4a6c3faed8ac93"} Nov 01 15:58:19.302916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.302858 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:19.305286 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.305263 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:19.319459 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.319389 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" podStartSLOduration=1.319376374 podStartE2EDuration="1.319376374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:19.318437276 +0000 UTC m=+637.021603228" watchObservedRunningTime="2026-11-01 15:58:19.319376374 +0000 UTC m=+637.022542310" Nov 01 15:58:19.342266 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.342122 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" podStartSLOduration=2.342100605 podStartE2EDuration="2.342100605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:19.340624829 +0000 UTC m=+637.043790768" watchObservedRunningTime="2026-11-01 15:58:19.342100605 +0000 UTC m=+637.045266542" Nov 01 15:58:19.395797 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.391573 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:19.496206 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.496169 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:19.599912 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:19.599883 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq"] Nov 01 15:58:19.601942 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:19.601916 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e04b490_5013_4ff0_bd43_c529b49a559f.slice/crio-762defcca9785600f7e55b4c1395c5209231e89b24fe1bd0d6bba9779c00876d WatchSource:0}: Error finding container 762defcca9785600f7e55b4c1395c5209231e89b24fe1bd0d6bba9779c00876d: Status 404 returned error can't find the container with id 762defcca9785600f7e55b4c1395c5209231e89b24fe1bd0d6bba9779c00876d Nov 01 15:58:20.035050 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.035016 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:20.129072 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.129037 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:20.154363 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.154334 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4"] Nov 01 15:58:20.158512 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.158482 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:20.163883 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.163742 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-dfff59\"" Nov 01 15:58:20.165972 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.165951 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4"] Nov 01 15:58:20.200817 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.200788 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4\" (UID: \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:20.200941 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.200874 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjbc8\" (UniqueName: \"kubernetes.io/projected/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-kube-api-access-gjbc8\") pod \"payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4\" (UID: \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:20.254474 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.254447 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-dfff59-dockercfg-kkdbp\"" Nov 01 15:58:20.302271 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.302184 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjbc8\" (UniqueName: \"kubernetes.io/projected/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-kube-api-access-gjbc8\") pod \"payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4\" (UID: \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:20.302432 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.302329 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4\" (UID: \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:20.303132 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.303106 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4\" (UID: \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:20.311846 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.311210 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" event={"ID":"2e04b490-5013-4ff0-bd43-c529b49a559f","Type":"ContainerStarted","Data":"85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510"} Nov 01 15:58:20.311846 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.311278 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" event={"ID":"2e04b490-5013-4ff0-bd43-c529b49a559f","Type":"ContainerStarted","Data":"762defcca9785600f7e55b4c1395c5209231e89b24fe1bd0d6bba9779c00876d"} Nov 01 15:58:20.311846 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.311535 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:20.314457 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.314432 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjbc8\" (UniqueName: \"kubernetes.io/projected/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-kube-api-access-gjbc8\") pod \"payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4\" (UID: \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:20.324214 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.323674 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:20.334684 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.334640 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" podStartSLOduration=2.334627275 podStartE2EDuration="2.334627275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:20.332956163 +0000 UTC m=+638.036122099" watchObservedRunningTime="2026-11-01 15:58:20.334627275 +0000 UTC m=+638.037793251" Nov 01 15:58:20.370978 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.370945 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9"] Nov 01 15:58:20.374730 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.374713 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:20.382003 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.381981 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-bd831a\"" Nov 01 15:58:20.382112 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.382074 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-bd831a-dockercfg-whrbm\"" Nov 01 15:58:20.385697 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.385668 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9"] Nov 01 15:58:20.428955 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.428921 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:20.479138 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.479110 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:20.525572 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.525526 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b94c21-751f-4484-9cfd-8100407cb40f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9\" (UID: \"c5b94c21-751f-4484-9cfd-8100407cb40f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:20.525792 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.525607 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mccmv\" (UniqueName: \"kubernetes.io/projected/c5b94c21-751f-4484-9cfd-8100407cb40f-kube-api-access-mccmv\") pod \"payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9\" (UID: \"c5b94c21-751f-4484-9cfd-8100407cb40f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:20.626680 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.626646 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b94c21-751f-4484-9cfd-8100407cb40f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9\" (UID: \"c5b94c21-751f-4484-9cfd-8100407cb40f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:20.626885 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.626695 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mccmv\" (UniqueName: \"kubernetes.io/projected/c5b94c21-751f-4484-9cfd-8100407cb40f-kube-api-access-mccmv\") pod \"payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9\" (UID: \"c5b94c21-751f-4484-9cfd-8100407cb40f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:20.627422 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.627401 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b94c21-751f-4484-9cfd-8100407cb40f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9\" (UID: \"c5b94c21-751f-4484-9cfd-8100407cb40f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:20.634278 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.634253 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mccmv\" (UniqueName: \"kubernetes.io/projected/c5b94c21-751f-4484-9cfd-8100407cb40f-kube-api-access-mccmv\") pod \"payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9\" (UID: \"c5b94c21-751f-4484-9cfd-8100407cb40f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:20.690613 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.690588 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:20.788313 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.788254 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4"] Nov 01 15:58:20.789974 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:20.789946 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad7ef24_f0a2_4901_8301_fb6e72ccfb81.slice/crio-f177124e6e609c4a7c3fb12552a708cd95c17aed8a46a3230f55b12a32335953 WatchSource:0}: Error finding container f177124e6e609c4a7c3fb12552a708cd95c17aed8a46a3230f55b12a32335953: Status 404 returned error can't find the container with id f177124e6e609c4a7c3fb12552a708cd95c17aed8a46a3230f55b12a32335953 Nov 01 15:58:20.813834 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.811667 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:20.901356 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.901298 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:20.974305 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:20.974231 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9"] Nov 01 15:58:20.976627 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:20.976598 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b94c21_751f_4484_9cfd_8100407cb40f.slice/crio-b48e7e2abf3ee4dbcea203a387db1000f5cbee1cd5c2cb3c70dc696147602005 WatchSource:0}: Error finding container b48e7e2abf3ee4dbcea203a387db1000f5cbee1cd5c2cb3c70dc696147602005: Status 404 returned error can't find the container with id b48e7e2abf3ee4dbcea203a387db1000f5cbee1cd5c2cb3c70dc696147602005 Nov 01 15:58:21.278542 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.278513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:21.319417 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.319382 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" event={"ID":"dad7ef24-f0a2-4901-8301-fb6e72ccfb81","Type":"ContainerStarted","Data":"c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763"} Nov 01 15:58:21.319602 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.319446 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" event={"ID":"dad7ef24-f0a2-4901-8301-fb6e72ccfb81","Type":"ContainerStarted","Data":"f177124e6e609c4a7c3fb12552a708cd95c17aed8a46a3230f55b12a32335953"} Nov 01 15:58:21.319602 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.319478 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:21.319602 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.319594 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh"] Nov 01 15:58:21.323615 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.323596 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:21.323731 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.323624 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" event={"ID":"c5b94c21-751f-4484-9cfd-8100407cb40f","Type":"ContainerStarted","Data":"c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa"} Nov 01 15:58:21.323731 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.323652 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:21.323731 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.323654 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" event={"ID":"c5b94c21-751f-4484-9cfd-8100407cb40f","Type":"ContainerStarted","Data":"b48e7e2abf3ee4dbcea203a387db1000f5cbee1cd5c2cb3c70dc696147602005"} Nov 01 15:58:21.334580 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.333534 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-6b0e0f\"" Nov 01 15:58:21.334580 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.333591 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-6b0e0f-dockercfg-nv5t6\"" Nov 01 15:58:21.334848 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.334825 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh"] Nov 01 15:58:21.339261 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.339236 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p"] Nov 01 15:58:21.340086 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.340014 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" podStartSLOduration=1.339998923 podStartE2EDuration="1.339998923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:21.339100066 +0000 UTC m=+639.042266006" watchObservedRunningTime="2026-11-01 15:58:21.339998923 +0000 UTC m=+639.043164860" Nov 01 15:58:21.343218 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.343201 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:21.355799 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.355775 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p"] Nov 01 15:58:21.370677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.370657 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:21.381186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.381143 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" podStartSLOduration=1.381130269 podStartE2EDuration="1.381130269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:21.379729354 +0000 UTC m=+639.082895288" watchObservedRunningTime="2026-11-01 15:58:21.381130269 +0000 UTC m=+639.084296221" Nov 01 15:58:21.434128 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.434095 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915836fc-5724-4713-a688-dc226a3e7818-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh\" (UID: \"915836fc-5724-4713-a688-dc226a3e7818\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:21.434282 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.434133 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfqg\" (UniqueName: \"kubernetes.io/projected/46336b63-f773-4697-a916-8adf998d6da2-kube-api-access-hdfqg\") pod \"payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p\" (UID: \"46336b63-f773-4697-a916-8adf998d6da2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:21.434282 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.434155 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srf64\" (UniqueName: \"kubernetes.io/projected/915836fc-5724-4713-a688-dc226a3e7818-kube-api-access-srf64\") pod \"payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh\" (UID: \"915836fc-5724-4713-a688-dc226a3e7818\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:21.434391 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.434297 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46336b63-f773-4697-a916-8adf998d6da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p\" (UID: \"46336b63-f773-4697-a916-8adf998d6da2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:21.535374 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.535280 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46336b63-f773-4697-a916-8adf998d6da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p\" (UID: \"46336b63-f773-4697-a916-8adf998d6da2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:21.535551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.535419 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915836fc-5724-4713-a688-dc226a3e7818-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh\" (UID: \"915836fc-5724-4713-a688-dc226a3e7818\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:21.535551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.535451 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfqg\" (UniqueName: \"kubernetes.io/projected/46336b63-f773-4697-a916-8adf998d6da2-kube-api-access-hdfqg\") pod \"payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p\" (UID: \"46336b63-f773-4697-a916-8adf998d6da2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:21.535551 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.535473 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srf64\" (UniqueName: \"kubernetes.io/projected/915836fc-5724-4713-a688-dc226a3e7818-kube-api-access-srf64\") pod \"payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh\" (UID: \"915836fc-5724-4713-a688-dc226a3e7818\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:21.536175 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.536146 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46336b63-f773-4697-a916-8adf998d6da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p\" (UID: \"46336b63-f773-4697-a916-8adf998d6da2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:21.536452 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.536429 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915836fc-5724-4713-a688-dc226a3e7818-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh\" (UID: \"915836fc-5724-4713-a688-dc226a3e7818\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:21.543147 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.543125 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfqg\" (UniqueName: \"kubernetes.io/projected/46336b63-f773-4697-a916-8adf998d6da2-kube-api-access-hdfqg\") pod \"payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p\" (UID: \"46336b63-f773-4697-a916-8adf998d6da2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:21.543242 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.543224 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srf64\" (UniqueName: \"kubernetes.io/projected/915836fc-5724-4713-a688-dc226a3e7818-kube-api-access-srf64\") pod \"payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh\" (UID: \"915836fc-5724-4713-a688-dc226a3e7818\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:21.644364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.644330 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:21.656680 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.656648 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:21.767263 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.767222 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:21.843065 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.842972 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s"] Nov 01 15:58:21.851517 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.851476 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:21.852375 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.852226 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s"] Nov 01 15:58:21.870462 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.870429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:21.905496 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.905468 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-9f4dc9\"" Nov 01 15:58:21.921342 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.921278 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh"] Nov 01 15:58:21.922776 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:21.922736 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod915836fc_5724_4713_a688_dc226a3e7818.slice/crio-031ac16301c435519f34626a9a557b1b8887eea83ad4becd8b04462b46822b88 WatchSource:0}: Error finding container 031ac16301c435519f34626a9a557b1b8887eea83ad4becd8b04462b46822b88: Status 404 returned error can't find the container with id 031ac16301c435519f34626a9a557b1b8887eea83ad4becd8b04462b46822b88 Nov 01 15:58:21.926579 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.926556 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-9f4dc9-dockercfg-vrlwn\"" Nov 01 15:58:21.944534 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.944443 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xtkl\" (UniqueName: \"kubernetes.io/projected/1676214c-b5c3-42ae-a139-e6f0098f23cd-kube-api-access-7xtkl\") pod \"payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s\" (UID: \"1676214c-b5c3-42ae-a139-e6f0098f23cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:21.944534 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.944498 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676214c-b5c3-42ae-a139-e6f0098f23cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s\" (UID: \"1676214c-b5c3-42ae-a139-e6f0098f23cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:21.973700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:21.973673 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p"] Nov 01 15:58:22.045849 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.045817 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xtkl\" (UniqueName: \"kubernetes.io/projected/1676214c-b5c3-42ae-a139-e6f0098f23cd-kube-api-access-7xtkl\") pod \"payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s\" (UID: \"1676214c-b5c3-42ae-a139-e6f0098f23cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:22.045987 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.045860 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676214c-b5c3-42ae-a139-e6f0098f23cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s\" (UID: \"1676214c-b5c3-42ae-a139-e6f0098f23cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:22.046557 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.046534 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676214c-b5c3-42ae-a139-e6f0098f23cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s\" (UID: \"1676214c-b5c3-42ae-a139-e6f0098f23cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:22.054324 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.054299 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xtkl\" (UniqueName: \"kubernetes.io/projected/1676214c-b5c3-42ae-a139-e6f0098f23cd-kube-api-access-7xtkl\") pod \"payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s\" (UID: \"1676214c-b5c3-42ae-a139-e6f0098f23cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:22.166262 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.166220 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:22.178186 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.178157 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:22.281938 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.281897 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:22.328512 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.328473 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" event={"ID":"46336b63-f773-4697-a916-8adf998d6da2","Type":"ContainerStarted","Data":"f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9"} Nov 01 15:58:22.328688 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.328570 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" event={"ID":"46336b63-f773-4697-a916-8adf998d6da2","Type":"ContainerStarted","Data":"32a3a45446f81d5d205452a0006715743041f0c8aea14726ae4268b6be97bf9d"} Nov 01 15:58:22.328819 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.328799 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:22.330813 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.330787 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" event={"ID":"915836fc-5724-4713-a688-dc226a3e7818","Type":"ContainerStarted","Data":"7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf"} Nov 01 15:58:22.330813 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.330832 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" event={"ID":"915836fc-5724-4713-a688-dc226a3e7818","Type":"ContainerStarted","Data":"031ac16301c435519f34626a9a557b1b8887eea83ad4becd8b04462b46822b88"} Nov 01 15:58:22.331032 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.331009 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:22.345446 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.345388 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" podStartSLOduration=1.345370604 podStartE2EDuration="1.345370604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:22.343676461 +0000 UTC m=+640.046842465" watchObservedRunningTime="2026-11-01 15:58:22.345370604 +0000 UTC m=+640.048536540" Nov 01 15:58:22.365996 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.365961 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:22.458038 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.456672 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" podStartSLOduration=1.456653865 podStartE2EDuration="1.456653865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:22.363561706 +0000 UTC m=+640.066727656" watchObservedRunningTime="2026-11-01 15:58:22.456653865 +0000 UTC m=+640.159819802" Nov 01 15:58:22.460204 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.460180 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:58:22.461619 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:22.461596 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s"] Nov 01 15:58:23.164254 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:23.164223 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:23.341868 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:23.341830 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" event={"ID":"1676214c-b5c3-42ae-a139-e6f0098f23cd","Type":"ContainerStarted","Data":"5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23"} Nov 01 15:58:23.342310 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:23.341889 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" event={"ID":"1676214c-b5c3-42ae-a139-e6f0098f23cd","Type":"ContainerStarted","Data":"c2085c781c3e3330a0015f26ae55f604233ad6af81d414b91404a8e7062d327e"} Nov 01 15:58:23.357484 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:23.357440 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" podStartSLOduration=2.357428228 podStartE2EDuration="2.357428228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:23.355862967 +0000 UTC m=+641.059028903" watchObservedRunningTime="2026-11-01 15:58:23.357428228 +0000 UTC m=+641.060594164" Nov 01 15:58:24.117775 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:24.108310 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:24.185774 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:24.185106 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:24.348974 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:24.348945 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:24.389835 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:58:24.382135 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:58:24.435929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:24.435876 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:24.893786 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:24.893721 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:24.997777 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:24.987531 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:25.401773 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:25.401234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:25.695459 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:25.695312 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:25.793185 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:25.793070 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:26.297776 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:26.296817 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:58:26.721774 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:26.721330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:26.810087 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:26.807324 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:26.905776 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:26.905027 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:27.656433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:27.656378 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:28.686775 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:28.686351 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:29.700889 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:29.700798 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:30.042948 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:30.042861 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:58:30.147119 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:30.147084 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:30.245581 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:30.245544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:30.314827 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:30.314698 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 15:58:30.314827 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:30.314802 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 15:58:31.267639 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:31.267593 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 15:58:31.268130 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:31.267692 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 15:58:31.325405 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:31.325378 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 15:58:31.395602 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:31.395564 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:31.494393 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:31.494361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:32.062917 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.062878 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:32.161526 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.161480 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:32.255866 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.255314 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:40030->10.133.0.43:15021: read: connection reset by peer" start-of-body= Nov 01 15:58:32.255866 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.255521 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": read tcp 10.133.0.2:40030->10.133.0.43:15021: read: connection reset by peer" Nov 01 15:58:32.266104 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.265984 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:32.334475 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.334406 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 15:58:32.421071 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.421004 4573 generic.go:356] "Generic (PLEG): container finished" podID="5747c189-9aad-474e-b650-79061531f01d" containerID="2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c" exitCode=0 Nov 01 15:58:32.421212 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.421072 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" event={"ID":"5747c189-9aad-474e-b650-79061531f01d","Type":"ContainerDied","Data":"2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c"} Nov 01 15:58:32.421465 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.421450 4573 scope.go:122] "RemoveContainer" containerID="2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c" Nov 01 15:58:32.422987 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.422959 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:32.423095 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.423031 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:32.423095 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:32.423065 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:33.089856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.086967 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:33.290828 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.290701 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g"] Nov 01 15:58:33.294234 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.294212 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.299790 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.299072 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"simulator-w0-4a083d-kserve-self-signed-certs\"" Nov 01 15:58:33.299790 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.299152 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"default-dockercfg-lhz8h\"" Nov 01 15:58:33.300803 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.300778 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g"] Nov 01 15:58:33.334792 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.334747 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:33.345257 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.345192 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 15:58:33.345378 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.345265 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 15:58:33.345492 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.345473 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 15:58:33.386775 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.384373 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"openshift-service-ca.crt\"" Nov 01 15:58:33.387885 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.387848 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"kube-root-ca.crt\"" Nov 01 15:58:33.400660 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.400632 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-model-cache\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.400807 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.400677 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-home\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.400807 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.400731 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m2r\" (UniqueName: \"kubernetes.io/projected/7062fcad-aa5b-4c09-b743-fbaa104863c6-kube-api-access-k7m2r\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.400997 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.400971 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-tmp-dir\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.401101 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.401082 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-dshm\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.401160 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.401118 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7062fcad-aa5b-4c09-b743-fbaa104863c6-tls-certs\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.428350 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.428325 4573 generic.go:356] "Generic (PLEG): container finished" podID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerID="36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a" exitCode=0 Nov 01 15:58:33.428513 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.428405 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" event={"ID":"59186df5-07c5-4dbc-b45d-d474e1514fe6","Type":"ContainerDied","Data":"36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a"} Nov 01 15:58:33.428871 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.428846 4573 scope.go:122] "RemoveContainer" containerID="36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a" Nov 01 15:58:33.430382 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.430349 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:33.430497 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.430432 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:33.430497 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.430476 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:58:33.430626 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.430503 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" event={"ID":"5747c189-9aad-474e-b650-79061531f01d","Type":"ContainerStarted","Data":"54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6"} Nov 01 15:58:33.502039 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.502002 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-dshm\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.502215 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.502040 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7062fcad-aa5b-4c09-b743-fbaa104863c6-tls-certs\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.502215 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.502109 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-model-cache\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.502215 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.502150 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-home\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.502391 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.502231 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m2r\" (UniqueName: \"kubernetes.io/projected/7062fcad-aa5b-4c09-b743-fbaa104863c6-kube-api-access-k7m2r\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.502391 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.502356 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-tmp-dir\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.502817 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.502791 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-tmp-dir\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.503079 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.503059 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-model-cache\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.503145 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.503130 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-home\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.505078 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.505058 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-dshm\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.506572 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.506547 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7062fcad-aa5b-4c09-b743-fbaa104863c6-tls-certs\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.525386 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.524177 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m2r\" (UniqueName: \"kubernetes.io/projected/7062fcad-aa5b-4c09-b743-fbaa104863c6-kube-api-access-k7m2r\") pod \"simulator-w0-4a083d-kserve-686b95b847-zcq4g\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.615068 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.614916 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:33.750501 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.750470 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:33.750672 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.750618 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 15:58:33.750733 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.750666 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 15:58:33.932579 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:33.932552 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g"] Nov 01 15:58:33.934337 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:33.934308 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7062fcad_aa5b_4c09_b743_fbaa104863c6.slice/crio-6425046e453a71a89d191082d459de0ab32a4e16a1d22be49ce13cf2e40abe40 WatchSource:0}: Error finding container 6425046e453a71a89d191082d459de0ab32a4e16a1d22be49ce13cf2e40abe40: Status 404 returned error can't find the container with id 6425046e453a71a89d191082d459de0ab32a4e16a1d22be49ce13cf2e40abe40 Nov 01 15:58:34.174797 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:34.174462 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:34.438162 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:34.438127 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" event={"ID":"59186df5-07c5-4dbc-b45d-d474e1514fe6","Type":"ContainerStarted","Data":"c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3"} Nov 01 15:58:34.439723 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:34.439693 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" event={"ID":"7062fcad-aa5b-4c09-b743-fbaa104863c6","Type":"ContainerStarted","Data":"f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0"} Nov 01 15:58:34.439859 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:34.439735 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" event={"ID":"7062fcad-aa5b-4c09-b743-fbaa104863c6","Type":"ContainerStarted","Data":"6425046e453a71a89d191082d459de0ab32a4e16a1d22be49ce13cf2e40abe40"} Nov 01 15:58:34.486600 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:34.486540 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podStartSLOduration=1.4865200920000001 podStartE2EDuration="1.486520092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:34.480743675 +0000 UTC m=+652.183909611" watchObservedRunningTime="2026-11-01 15:58:34.486520092 +0000 UTC m=+652.189686029" Nov 01 15:58:34.615669 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:34.615632 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:34.749823 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:34.749674 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 15:58:34.749823 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:34.749723 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 15:58:35.299926 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:35.299887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:35.356948 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:35.356916 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 15:58:35.400777 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:35.399420 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:35.400777 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:35.399527 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 15:58:35.400777 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:35.399568 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 15:58:35.750013 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:35.749981 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 15:58:35.750434 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:35.750031 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 15:58:36.400776 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:36.397223 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 15:58:36.400776 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:36.397299 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 15:58:36.717356 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:36.717276 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:36.749968 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:36.749934 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 15:58:36.750175 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:36.749990 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 15:58:36.823693 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:36.823638 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:37.346650 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.346614 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:37.405773 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.405138 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 15:58:37.405773 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.405198 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 15:58:37.454309 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.454279 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:37.546495 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.546454 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:37.693272 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.693207 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:37.749541 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.749511 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Nov 01 15:58:37.750285 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.750256 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Nov 01 15:58:37.924433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:37.924385 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:38.031681 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:38.031556 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:38.397346 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:38.397311 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 15:58:38.397622 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:38.397588 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 15:58:38.399972 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:38.399944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:38.496881 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:38.496842 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:38.602916 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:38.602882 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:38.714877 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:38.714789 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:39.273137 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:39.273105 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:39.275008 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:39.274988 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:39.287771 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:39.283954 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 15:58:39.395272 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:39.395241 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Nov 01 15:58:39.395438 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:39.395293 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Nov 01 15:58:39.586832 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:39.586136 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:40.398581 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:40.398553 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:40.399057 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:40.398843 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:40.399961 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:40.399941 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 15:58:40.423328 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:40.423297 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:40.677146 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:40.677072 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:58:41.586260 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:41.586231 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:41.687317 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:41.687273 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:41.890338 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:41.890307 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:42.276560 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:42.276474 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:42.367734 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:42.367699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:42.481141 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:42.481106 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:42.575647 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:42.575574 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:42.676602 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:42.676567 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:42.932862 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:42.932828 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:43.043384 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:43.043350 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:43.523409 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:43.523377 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:43.615801 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:43.615767 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:43.628696 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:43.628668 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:43.631428 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:43.631395 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:43.727440 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:43.727412 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:44.531442 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:44.531410 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 15:58:44.560637 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:44.560604 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:44.695899 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:44.695864 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:45.494221 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:45.494192 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:46.483258 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:46.483225 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:47.271146 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.271111 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:47.547623 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.547550 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q"] Nov 01 15:58:47.553256 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.553235 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.561943 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.561921 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"kube-root-ca.crt\"" Nov 01 15:58:47.562055 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.562022 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"openshift-service-ca.crt\"" Nov 01 15:58:47.562781 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.562762 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"default-dockercfg-2f26v\"" Nov 01 15:58:47.562781 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.562771 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-6b0e0f\"/\"simulator-w2-6b0e0f-kserve-self-signed-certs\"" Nov 01 15:58:47.568802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.568774 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q"] Nov 01 15:58:47.580237 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.580215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:47.677297 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.677266 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1438f4f8-2770-4839-bfe2-b2023337521e-tls-certs\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.677297 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.677297 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-dshm\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.677483 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.677452 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfpj\" (UniqueName: \"kubernetes.io/projected/1438f4f8-2770-4839-bfe2-b2023337521e-kube-api-access-hpfpj\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.677520 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.677484 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-model-cache\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.677520 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.677502 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-home\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.677610 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.677593 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-tmp-dir\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.778665 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.778631 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfpj\" (UniqueName: \"kubernetes.io/projected/1438f4f8-2770-4839-bfe2-b2023337521e-kube-api-access-hpfpj\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.778665 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.778673 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-model-cache\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.778975 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.778695 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-home\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.778975 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.778730 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-tmp-dir\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.778975 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.778820 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1438f4f8-2770-4839-bfe2-b2023337521e-tls-certs\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.778975 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.778846 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-dshm\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.779305 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.779165 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-model-cache\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.779374 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.779350 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-tmp-dir\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.779430 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.779367 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-home\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.781236 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.781209 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-dshm\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.781894 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.781874 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1438f4f8-2770-4839-bfe2-b2023337521e-tls-certs\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.786468 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.786443 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfpj\" (UniqueName: \"kubernetes.io/projected/1438f4f8-2770-4839-bfe2-b2023337521e-kube-api-access-hpfpj\") pod \"simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:47.865250 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:47.865211 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:48.076918 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.076881 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:48.142616 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:48.142588 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1438f4f8_2770_4839_bfe2_b2023337521e.slice/crio-ab5b76041805e1765c0085a857adf952b36ef7111099f13e2bd50617d8291a88 WatchSource:0}: Error finding container ab5b76041805e1765c0085a857adf952b36ef7111099f13e2bd50617d8291a88: Status 404 returned error can't find the container with id ab5b76041805e1765c0085a857adf952b36ef7111099f13e2bd50617d8291a88 Nov 01 15:58:48.144973 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.144942 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q"] Nov 01 15:58:48.542199 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.542121 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" event={"ID":"1438f4f8-2770-4839-bfe2-b2023337521e","Type":"ContainerStarted","Data":"86e7944a47b83111cc1e7db4823b6d22fc3d154d83f6d41ce049ff8da9583a1b"} Nov 01 15:58:48.542199 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.542162 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" event={"ID":"1438f4f8-2770-4839-bfe2-b2023337521e","Type":"ContainerStarted","Data":"ab5b76041805e1765c0085a857adf952b36ef7111099f13e2bd50617d8291a88"} Nov 01 15:58:48.561129 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.561085 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podStartSLOduration=1.5610723370000001 podStartE2EDuration="1.561072337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:48.558481472 +0000 UTC m=+666.261647410" watchObservedRunningTime="2026-11-01 15:58:48.561072337 +0000 UTC m=+666.264238273" Nov 01 15:58:48.677617 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.677589 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9"] Nov 01 15:58:48.681509 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.681489 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.687526 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.687504 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"kube-root-ca.crt\"" Nov 01 15:58:48.687668 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.687547 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"openshift-service-ca.crt\"" Nov 01 15:58:48.687668 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.687621 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"simulator-w3-bd831a-kserve-self-signed-certs\"" Nov 01 15:58:48.687808 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.687690 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"default-dockercfg-bxf27\"" Nov 01 15:58:48.693605 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.693584 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9"] Nov 01 15:58:48.719599 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.719569 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:48.786773 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.786739 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tls-certs\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.786929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.786783 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tmp-dir\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.786929 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.786813 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-model-cache\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.787075 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.787052 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-home\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.787131 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.787101 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-dshm\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.787220 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.787202 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rjn5\" (UniqueName: \"kubernetes.io/projected/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-kube-api-access-5rjn5\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.865519 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.865492 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:48.888232 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888205 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-home\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.888399 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888251 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-dshm\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.888399 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888313 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rjn5\" (UniqueName: \"kubernetes.io/projected/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-kube-api-access-5rjn5\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.888399 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888376 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tls-certs\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.888565 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888410 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tmp-dir\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.888565 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888447 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-model-cache\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.888726 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888706 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-home\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.888928 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888910 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tmp-dir\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.889022 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.888998 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-model-cache\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.891154 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.891127 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-dshm\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.891265 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.891246 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tls-certs\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.899084 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.899063 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rjn5\" (UniqueName: \"kubernetes.io/projected/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-kube-api-access-5rjn5\") pod \"simulator-w3-bd831a-kserve-676dc75c6f-4w6b9\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:48.916429 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.916404 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:48.994096 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:48.994061 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:49.271723 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:49.271703 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9"] Nov 01 15:58:49.274126 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:58:49.274095 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f3d46b3_cd7c_4340_a37e_3571eb7b5892.slice/crio-5883ea9816e05df2976a32048623b26c891d44bc040c5d461bc2a35e3a7f7c93 WatchSource:0}: Error finding container 5883ea9816e05df2976a32048623b26c891d44bc040c5d461bc2a35e3a7f7c93: Status 404 returned error can't find the container with id 5883ea9816e05df2976a32048623b26c891d44bc040c5d461bc2a35e3a7f7c93 Nov 01 15:58:49.548364 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:49.548277 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:58:49.549960 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:49.549929 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" event={"ID":"0f3d46b3-cd7c-4340-a37e-3571eb7b5892","Type":"ContainerStarted","Data":"9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587"} Nov 01 15:58:49.550087 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:49.549975 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" event={"ID":"0f3d46b3-cd7c-4340-a37e-3571eb7b5892","Type":"ContainerStarted","Data":"5883ea9816e05df2976a32048623b26c891d44bc040c5d461bc2a35e3a7f7c93"} Nov 01 15:58:49.568829 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:49.568776 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podStartSLOduration=1.568746386 podStartE2EDuration="1.568746386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:58:49.565857526 +0000 UTC m=+667.269023461" watchObservedRunningTime="2026-11-01 15:58:49.568746386 +0000 UTC m=+667.271912323" Nov 01 15:58:49.771369 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:49.771335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:49.994285 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:49.994252 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:50.759799 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:50.759760 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:51.394671 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:51.394640 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:51.522790 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:51.522739 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:52.255985 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:52.255953 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:52.431873 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:52.431840 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:53.070666 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:53.070639 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:53.857652 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:53.857577 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:54.364928 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:58:54.364881 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:58:54.671464 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:54.671391 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:55.308236 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:55.308180 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:55.435630 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:55.435596 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:56.463471 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:56.463439 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:57.266280 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:57.266247 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:57.866174 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:57.866139 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:57.878683 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:57.878657 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:57.959145 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:57.959119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:58.611950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:58.611923 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 15:58:58.759646 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:58.759618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:58.994975 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:58.994873 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:59.007590 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:59.007564 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:58:59.445366 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:59.445335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:58:59.616722 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:58:59.616689 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 15:59:00.079082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:00.079049 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:00.775690 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:00.775662 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:01.044607 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:01.044519 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:59:01.684157 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:01.684120 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:02.157441 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.157404 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87"] Nov 01 15:59:02.161505 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.161483 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.168313 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.168289 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-4a083d\"/\"premium-w0-4a083d-kserve-self-signed-certs\"" Nov 01 15:59:02.172960 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.172933 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87"] Nov 01 15:59:02.215952 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.215919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:02.330469 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.330432 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp9fv\" (UniqueName: \"kubernetes.io/projected/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-kube-api-access-xp9fv\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.330642 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.330490 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-dshm\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.330642 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.330605 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tmp-dir\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.330725 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.330670 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-home\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.330725 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.330718 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-model-cache\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.330815 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.330800 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tls-certs\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.431937 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.431852 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp9fv\" (UniqueName: \"kubernetes.io/projected/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-kube-api-access-xp9fv\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.431937 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.431907 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-dshm\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.431937 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.431938 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tmp-dir\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.432222 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.431966 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-home\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.432222 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.431999 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-model-cache\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.432222 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.432049 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tls-certs\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.432504 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.432480 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-home\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.432641 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.432481 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tmp-dir\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.432641 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.432571 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-model-cache\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.434209 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.434189 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-dshm\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.434684 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.434664 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tls-certs\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.439646 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.439602 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp9fv\" (UniqueName: \"kubernetes.io/projected/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-kube-api-access-xp9fv\") pod \"premium-w0-4a083d-kserve-67695754f8-2mw87\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.444582 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.444552 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:02.477063 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.477037 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:02.763800 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:02.763769 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87"] Nov 01 15:59:02.764292 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:59:02.764269 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a7f0f44_adab_46e3_9232_a1a4f3a30e6e.slice/crio-b00e462a37503078ff99e806d1f515516a482cff7ae0dea011b4aa5c80ffd707 WatchSource:0}: Error finding container b00e462a37503078ff99e806d1f515516a482cff7ae0dea011b4aa5c80ffd707: Status 404 returned error can't find the container with id b00e462a37503078ff99e806d1f515516a482cff7ae0dea011b4aa5c80ffd707 Nov 01 15:59:03.112682 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:03.112648 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:03.630284 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:03.630253 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" event={"ID":"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e","Type":"ContainerStarted","Data":"f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc"} Nov 01 15:59:03.630492 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:03.630293 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" event={"ID":"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e","Type":"ContainerStarted","Data":"b00e462a37503078ff99e806d1f515516a482cff7ae0dea011b4aa5c80ffd707"} Nov 01 15:59:03.649109 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:03.649064 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podStartSLOduration=1.649050495 podStartE2EDuration="1.649050495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:03.646895167 +0000 UTC m=+681.350061103" watchObservedRunningTime="2026-11-01 15:59:03.649050495 +0000 UTC m=+681.352216431" Nov 01 15:59:03.822304 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:03.822272 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:04.477525 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:04.477491 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:04.608821 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:04.608777 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:05.294007 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:05.293977 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:05.968113 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:05.968069 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:06.745256 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:06.745220 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:07.485802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:07.485772 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:08.474084 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:08.474055 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:09.138457 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:09.138426 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:09.232092 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:09.232050 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:10.060558 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:10.060526 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:10.852393 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:10.852342 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:11.581391 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:11.581363 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:12.361168 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:12.361138 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:12.477743 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:12.477709 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:12.490118 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:12.490091 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:12.664121 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:12.664038 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:59:12.690353 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:12.690331 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 15:59:13.086441 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.086357 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:13.273190 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.273144 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:13.806045 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.806013 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq"] Nov 01 15:59:13.810025 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.810005 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.819321 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.819294 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9f4dc9-openshift-default-dockercfg-tmp6r\"" Nov 01 15:59:13.823237 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.823211 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq"] Nov 01 15:59:13.848884 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.848857 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.849023 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.848888 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.849094 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.849050 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.849150 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.849091 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.849208 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.849180 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.849275 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.849255 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.849345 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.849328 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.849387 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.849367 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wsm\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-kube-api-access-s7wsm\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.849449 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.849436 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c25e01d-b4b7-4198-a519-d1c6614e8893-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.950593 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950565 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.950782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950599 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.950782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950621 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.950782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950678 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.950782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950709 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.950782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950777 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wsm\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-kube-api-access-s7wsm\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.951069 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950830 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c25e01d-b4b7-4198-a519-d1c6614e8893-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.951069 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950886 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.951069 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.950910 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.951069 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.951053 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.951272 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.951113 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.951272 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.951195 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.951374 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.951325 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.951505 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.951484 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c25e01d-b4b7-4198-a519-d1c6614e8893-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.952917 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.952894 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.953441 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.953420 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.959306 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.959279 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:13.960164 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:13.960140 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wsm\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-kube-api-access-s7wsm\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:14.078879 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.078787 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:14.125853 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.125811 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:14.416094 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.416067 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq"] Nov 01 15:59:14.417501 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:59:14.417473 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c25e01d_b4b7_4198_a519_d1c6614e8893.slice/crio-ef15d2f298d5d4fa4089150c7f818a2c8c2776da5e5a492fab71cfa04e4204b3 WatchSource:0}: Error finding container ef15d2f298d5d4fa4089150c7f818a2c8c2776da5e5a492fab71cfa04e4204b3: Status 404 returned error can't find the container with id ef15d2f298d5d4fa4089150c7f818a2c8c2776da5e5a492fab71cfa04e4204b3 Nov 01 15:59:14.420042 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.420009 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:14.420128 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.420074 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:14.420128 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.420105 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:14.690067 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.689990 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9"] Nov 01 15:59:14.694059 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.694032 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.694542 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.694509 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" event={"ID":"9c25e01d-b4b7-4198-a519-d1c6614e8893","Type":"ContainerStarted","Data":"55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b"} Nov 01 15:59:14.694651 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.694586 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" event={"ID":"9c25e01d-b4b7-4198-a519-d1c6614e8893","Type":"ContainerStarted","Data":"ef15d2f298d5d4fa4089150c7f818a2c8c2776da5e5a492fab71cfa04e4204b3"} Nov 01 15:59:14.701571 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.701543 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bd831a\"/\"premium-w3-bd831a-kserve-self-signed-certs\"" Nov 01 15:59:14.701696 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.701674 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9"] Nov 01 15:59:14.725330 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.725302 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:14.731636 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.731568 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podStartSLOduration=1.7315515179999998 podStartE2EDuration="1.731551518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:14.73121317 +0000 UTC m=+692.434379110" watchObservedRunningTime="2026-11-01 15:59:14.731551518 +0000 UTC m=+692.434717459" Nov 01 15:59:14.758969 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.758936 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tmp-dir\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.759093 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.758971 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-home\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.759700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.759228 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-dshm\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.759700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.759282 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-model-cache\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.759700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.759463 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tgvg\" (UniqueName: \"kubernetes.io/projected/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-kube-api-access-8tgvg\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.759700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.759570 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tls-certs\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.861801 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.861395 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-dshm\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.862400 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.861994 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-model-cache\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.862400 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.862079 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tgvg\" (UniqueName: \"kubernetes.io/projected/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-kube-api-access-8tgvg\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.862400 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.862156 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tls-certs\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.862400 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.862228 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tmp-dir\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.862400 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.862256 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-home\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.862718 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.862688 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-home\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.863131 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.863101 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tmp-dir\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.863240 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.863154 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-model-cache\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.865657 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.865635 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-dshm\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.866784 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.866060 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tls-certs\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:14.877772 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.876857 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:14.891807 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:14.886248 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tgvg\" (UniqueName: \"kubernetes.io/projected/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-kube-api-access-8tgvg\") pod \"premium-w3-bd831a-kserve-7445b5dc84-nc8w9\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:15.009126 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.009027 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:15.126742 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.126687 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:15.127947 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.127919 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:59:15.128082 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.127972 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 01 15:59:15.341188 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.341158 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9"] Nov 01 15:59:15.342727 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:59:15.342699 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47dfa29a_24f6_40a0_8505_be7c5f4a36c3.slice/crio-4a13dd87e4daf887863991abd71f6cc6fe8dfb7f3c86f7e570ffd5a0efaec1b5 WatchSource:0}: Error finding container 4a13dd87e4daf887863991abd71f6cc6fe8dfb7f3c86f7e570ffd5a0efaec1b5: Status 404 returned error can't find the container with id 4a13dd87e4daf887863991abd71f6cc6fe8dfb7f3c86f7e570ffd5a0efaec1b5 Nov 01 15:59:15.611412 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.611379 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:15.704541 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.704501 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" event={"ID":"47dfa29a-24f6-40a0-8505-be7c5f4a36c3","Type":"ContainerStarted","Data":"a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c"} Nov 01 15:59:15.704713 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.704552 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" event={"ID":"47dfa29a-24f6-40a0-8505-be7c5f4a36c3","Type":"ContainerStarted","Data":"4a13dd87e4daf887863991abd71f6cc6fe8dfb7f3c86f7e570ffd5a0efaec1b5"} Nov 01 15:59:15.722194 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:15.722135 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podStartSLOduration=1.722120506 podStartE2EDuration="1.722120506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:15.720847499 +0000 UTC m=+693.424013435" watchObservedRunningTime="2026-11-01 15:59:15.722120506 +0000 UTC m=+693.425286442" Nov 01 15:59:16.009223 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:16.009137 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:16.126826 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:16.126796 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:59:16.126992 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:16.126844 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 01 15:59:16.324766 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:16.324659 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:16.887530 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:16.887493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:17.105229 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:17.105195 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:17.126710 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:17.126682 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:59:17.126872 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:17.126732 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 01 15:59:17.651608 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:17.651578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:17.884782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:17.883189 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:18.126406 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:18.126373 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:59:18.126817 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:18.126424 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 01 15:59:18.834521 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:18.834482 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:19.126361 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:19.126328 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:59:19.126572 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:19.126380 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 01 15:59:19.556994 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:19.556909 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:20.126432 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:20.126400 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Nov 01 15:59:20.126639 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:20.126460 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Nov 01 15:59:20.372985 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:20.372946 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:21.131166 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:21.131138 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:21.131619 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:21.131420 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:21.132319 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:21.132298 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 15:59:21.363654 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:21.363620 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:22.078035 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:22.077998 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:22.874382 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:22.874351 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:23.574859 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:23.574820 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:23.684400 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:23.684364 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:24.271372 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:24.271341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:24.365644 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:59:24.365611 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:59:25.009511 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:25.009480 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:25.022112 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:25.022089 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:25.043057 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:25.043034 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:25.766224 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:25.766194 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:25.774052 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:25.774024 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 15:59:26.579993 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:26.579962 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:27.634969 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:27.634915 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:28.494313 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:28.494274 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:29.033260 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.033219 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr"] Nov 01 15:59:29.037310 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.037284 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.046562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.046538 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bd831a-openshift-default-dockercfg-f9pt7\"" Nov 01 15:59:29.049587 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.049564 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr"] Nov 01 15:59:29.136401 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136369 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.136544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136408 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.136544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136443 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.136544 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136488 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.136677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136580 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.136677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136615 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r4jf\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-kube-api-access-6r4jf\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.136677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136638 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.136677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136653 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.136859 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.136810 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.237909 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.237876 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.237909 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.237913 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238151 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.237939 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238151 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.237991 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238151 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238081 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238151 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238122 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r4jf\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-kube-api-access-6r4jf\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238357 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238159 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238357 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238188 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238357 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238278 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238357 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238292 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238549 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238485 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238549 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238531 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238656 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238610 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.238898 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.238879 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.240293 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.240266 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.240813 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.240793 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.246535 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.246510 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.246697 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.246677 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r4jf\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-kube-api-access-6r4jf\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.339646 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.339578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:29.351592 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.351569 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:29.637744 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.637716 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr"] Nov 01 15:59:29.639163 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:59:29.639137 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ad92a9_1bc0_4c24_a549_d76ab7e8419b.slice/crio-b3603f555f07fa8fa950948b34d704cacdb4533f913e8ed859c8d2d1cba439ea WatchSource:0}: Error finding container b3603f555f07fa8fa950948b34d704cacdb4533f913e8ed859c8d2d1cba439ea: Status 404 returned error can't find the container with id b3603f555f07fa8fa950948b34d704cacdb4533f913e8ed859c8d2d1cba439ea Nov 01 15:59:29.641473 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.641442 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:29.641562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.641522 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:29.641562 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.641553 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:29.787277 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.787237 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" event={"ID":"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b","Type":"ContainerStarted","Data":"f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7"} Nov 01 15:59:29.787456 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.787298 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" event={"ID":"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b","Type":"ContainerStarted","Data":"b3603f555f07fa8fa950948b34d704cacdb4533f913e8ed859c8d2d1cba439ea"} Nov 01 15:59:29.808942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:29.808890 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podStartSLOduration=0.808876684 podStartE2EDuration="808.876684ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:29.805556594 +0000 UTC m=+707.508722542" watchObservedRunningTime="2026-11-01 15:59:29.808876684 +0000 UTC m=+707.512042620" Nov 01 15:59:30.062856 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:30.062776 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:30.165362 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:30.165328 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:30.352698 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:30.352609 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:30.354320 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:30.354292 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:59:30.354444 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:30.354348 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 15:59:30.784773 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:30.762427 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:31.352447 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:31.352411 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:59:31.352848 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:31.352461 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 15:59:31.377780 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:31.373793 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:59:31.844270 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:31.844237 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:32.352053 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:32.352022 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:59:32.352236 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:32.352075 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 15:59:32.978635 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:32.978590 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:33.352604 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:33.352508 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:59:33.352604 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:33.352573 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 15:59:33.785652 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:33.785618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:34.352594 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:34.352563 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:59:34.353018 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:34.352613 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 15:59:34.681801 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:34.681708 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:35.352189 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:35.352155 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 15:59:35.352361 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:35.352208 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 15:59:35.522219 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:35.522186 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:36.470600 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:36.470563 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:36.560700 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:36.560668 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:36.561227 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:36.561208 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:36.562070 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:36.562053 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 15:59:37.266560 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:37.266525 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:38.169482 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:38.169451 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:38.778274 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:38.778241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:39.466998 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:39.466960 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:40.157521 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:40.157491 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:40.852343 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:40.852311 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:41.474290 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:41.474252 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:42.151947 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:42.151919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:42.779034 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:42.778997 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:43.555743 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.555710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:43.709900 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.709866 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22"] Nov 01 15:59:43.715105 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.715086 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.723159 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.723131 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-dfff59-openshift-default-dockercfg-bzb6m\"" Nov 01 15:59:43.728003 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.727977 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22"] Nov 01 15:59:43.732613 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.732589 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.732742 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.732619 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.732742 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.732640 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.732742 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.732655 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts7c9\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-kube-api-access-ts7c9\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.732921 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.732898 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.732975 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.732938 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.733024 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.732975 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.733092 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.733075 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90da294e-0e4c-4a46-920c-3f1df5e3343c-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.733216 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.733200 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834259 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834179 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90da294e-0e4c-4a46-920c-3f1df5e3343c-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834435 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834351 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834508 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834445 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834508 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834479 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834637 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834509 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834637 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834535 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts7c9\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-kube-api-access-ts7c9\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834637 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834609 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834638 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834802 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834677 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.834943 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834888 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.835079 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.835054 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.835143 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.835014 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90da294e-0e4c-4a46-920c-3f1df5e3343c-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.835143 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.834957 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.835255 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.835232 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.836831 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.836809 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.836921 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.836845 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.845267 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.845245 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:43.845474 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:43.845459 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts7c9\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-kube-api-access-ts7c9\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:44.029950 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.029916 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:44.257357 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.257320 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:44.319222 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.318325 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22"] Nov 01 15:59:44.333142 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:59:44.333086 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90da294e_0e4c_4a46_920c_3f1df5e3343c.slice/crio-6fcd927be0c5a1b0c57813534c30aa4f4b8af2aa3132e96fa78b2bb21071d068 WatchSource:0}: Error finding container 6fcd927be0c5a1b0c57813534c30aa4f4b8af2aa3132e96fa78b2bb21071d068: Status 404 returned error can't find the container with id 6fcd927be0c5a1b0c57813534c30aa4f4b8af2aa3132e96fa78b2bb21071d068 Nov 01 15:59:44.335419 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.335381 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:44.335506 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.335472 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:44.335569 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.335515 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:44.890488 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.890445 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" event={"ID":"90da294e-0e4c-4a46-920c-3f1df5e3343c","Type":"ContainerStarted","Data":"b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae"} Nov 01 15:59:44.891163 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.890512 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" event={"ID":"90da294e-0e4c-4a46-920c-3f1df5e3343c","Type":"ContainerStarted","Data":"6fcd927be0c5a1b0c57813534c30aa4f4b8af2aa3132e96fa78b2bb21071d068"} Nov 01 15:59:44.910677 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:44.910626 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podStartSLOduration=1.910612842 podStartE2EDuration="1.910612842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 15:59:44.908059592 +0000 UTC m=+722.611225541" watchObservedRunningTime="2026-11-01 15:59:44.910612842 +0000 UTC m=+722.613778778" Nov 01 15:59:45.030439 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:45.030407 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:45.031815 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:45.031789 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:59:45.031940 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:45.031838 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 15:59:45.052623 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:45.052592 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:45.784774 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:45.784338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:46.030554 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:46.030517 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:59:46.030984 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:46.030575 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 15:59:46.615107 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:46.615067 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:47.030238 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:47.030153 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:59:47.030238 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:47.030200 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 15:59:47.631397 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:47.631360 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:48.030858 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:48.030740 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:59:48.031182 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:48.031147 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 15:59:48.487998 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:48.487955 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:49.030843 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:49.030802 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:59:49.031260 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:49.030873 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 15:59:49.294371 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:49.294293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:50.030568 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:50.030533 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 01 15:59:50.030788 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:50.030586 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 01 15:59:50.552796 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:50.552765 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:51.569528 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:51.569492 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:51.669160 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:51.669105 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:51.669587 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:51.669560 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:51.670814 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:51.670791 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 15:59:52.447482 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:52.447445 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:53.264816 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:53.264740 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:53.964938 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:53.964905 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:54.364172 ip-10-0-141-145 kubenswrapper[4573]: E1101 15:59:54.364134 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 15:59:54.652317 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:54.652211 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:55.371715 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:55.371668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:56.049088 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:56.049049 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:56.739389 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:56.739336 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:57.417693 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:57.417656 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:58.187859 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.187825 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 15:59:58.252984 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.252943 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:58.945234 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.945197 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7"] Nov 01 15:59:58.949188 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.949163 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.957310 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.957282 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-6b0e0f-openshift-default-dockercfg-bgk88\"" Nov 01 15:59:58.960736 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.960713 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7"] Nov 01 15:59:58.982722 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.982695 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.982893 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.982733 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.982893 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.982871 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.982983 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.982951 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.982983 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.982968 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.983069 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.982994 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9a4c9-737d-4373-b1ee-aec239b0803e-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.983069 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.983029 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg96w\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-kube-api-access-xg96w\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.983157 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.983084 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:58.983157 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:58.983114 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.064232 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.064171 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 15:59:59.083777 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.083730 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.083961 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.083793 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.083961 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.083829 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9a4c9-737d-4373-b1ee-aec239b0803e-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084092 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.083956 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg96w\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-kube-api-access-xg96w\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084092 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084040 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084092 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084083 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084259 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084132 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084259 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084236 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084377 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084275 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084377 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084345 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084377 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084352 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084595 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084568 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084690 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084625 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.084942 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.084892 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9a4c9-737d-4373-b1ee-aec239b0803e-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.086256 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.086237 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.086776 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.086743 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.091348 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.091328 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.091433 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.091394 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg96w\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-kube-api-access-xg96w\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.266782 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.266664 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 15:59:59.552689 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.552659 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7"] Nov 01 15:59:59.554538 ip-10-0-141-145 kubenswrapper[4573]: W1101 15:59:59.554510 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36b9a4c9_737d_4373_b1ee_aec239b0803e.slice/crio-fdb4b04e0df25defb8f7e1bf8c38a527a62bd32027dc24561d49e35675d4e13a WatchSource:0}: Error finding container fdb4b04e0df25defb8f7e1bf8c38a527a62bd32027dc24561d49e35675d4e13a: Status 404 returned error can't find the container with id fdb4b04e0df25defb8f7e1bf8c38a527a62bd32027dc24561d49e35675d4e13a Nov 01 15:59:59.556998 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.556965 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:59.557113 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.557030 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:59.557113 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.557059 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 15:59:59.752252 ip-10-0-141-145 kubenswrapper[4573]: I1101 15:59:59.752210 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:00.002346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.002307 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" event={"ID":"36b9a4c9-737d-4373-b1ee-aec239b0803e","Type":"ContainerStarted","Data":"c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8"} Nov 01 16:00:00.002567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.002383 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" event={"ID":"36b9a4c9-737d-4373-b1ee-aec239b0803e","Type":"ContainerStarted","Data":"fdb4b04e0df25defb8f7e1bf8c38a527a62bd32027dc24561d49e35675d4e13a"} Nov 01 16:00:00.024204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.024156 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podStartSLOduration=2.024142497 podStartE2EDuration="2.024142497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 15:59:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:00.021492324 +0000 UTC m=+737.724658262" watchObservedRunningTime="2026-11-01 16:00:00.024142497 +0000 UTC m=+737.727308432" Nov 01 16:00:00.142489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.142459 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5"] Nov 01 16:00:00.146382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.146362 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9"] Nov 01 16:00:00.146525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.146503 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:00.150085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.150062 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw"] Nov 01 16:00:00.150231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.150186 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:00.153500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.153447 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:00:00.154409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.154377 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r"] Nov 01 16:00:00.154625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.154605 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:00.159132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.159097 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw"] Nov 01 16:00:00.159241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.159205 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:00.162881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.162860 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5"] Nov 01 16:00:00.165248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.165224 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r"] Nov 01 16:00:00.168638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.168619 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6"] Nov 01 16:00:00.173679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.173651 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:00.180909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.180883 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9"] Nov 01 16:00:00.187764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.187723 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6"] Nov 01 16:00:00.198309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198281 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm8j5\" (UniqueName: \"kubernetes.io/projected/de4b893c-251a-445b-b9e7-a1c68571497d-kube-api-access-hm8j5\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6\" (UID: \"de4b893c-251a-445b-b9e7-a1c68571497d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:00.198429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198314 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1605fbbc-8000-4cab-a969-b1a6eef41b6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9\" (UID: \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:00.198508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198485 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz4rk\" (UniqueName: \"kubernetes.io/projected/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-kube-api-access-rz4rk\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw\" (UID: \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:00.198550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198529 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875s2\" (UniqueName: \"kubernetes.io/projected/c08b4081-5132-4dc8-95b8-df8769d19e57-kube-api-access-875s2\") pod \"maas-api-key-cleanup-29892480-rqwd5\" (UID: \"c08b4081-5132-4dc8-95b8-df8769d19e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:00.198615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198602 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4b893c-251a-445b-b9e7-a1c68571497d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6\" (UID: \"de4b893c-251a-445b-b9e7-a1c68571497d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:00.198651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198640 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08b4081-5132-4dc8-95b8-df8769d19e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892480-rqwd5\" (UID: \"c08b4081-5132-4dc8-95b8-df8769d19e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:00.198688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198673 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchkq\" (UniqueName: \"kubernetes.io/projected/1605fbbc-8000-4cab-a969-b1a6eef41b6d-kube-api-access-gchkq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9\" (UID: \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:00.198809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198785 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3469177a-3810-4974-b737-66199dff79fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r\" (UID: \"3469177a-3810-4974-b737-66199dff79fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:00.198940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198845 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw\" (UID: \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:00.198940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.198910 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldh9\" (UniqueName: \"kubernetes.io/projected/3469177a-3810-4974-b737-66199dff79fd-kube-api-access-jldh9\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r\" (UID: \"3469177a-3810-4974-b737-66199dff79fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:00.272833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.270159 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 16:00:00.285026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.284977 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 16:00:00.285277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.285061 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 16:00:00.300240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300205 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm8j5\" (UniqueName: \"kubernetes.io/projected/de4b893c-251a-445b-b9e7-a1c68571497d-kube-api-access-hm8j5\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6\" (UID: \"de4b893c-251a-445b-b9e7-a1c68571497d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:00.300240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300240 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1605fbbc-8000-4cab-a969-b1a6eef41b6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9\" (UID: \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:00.300460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300310 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz4rk\" (UniqueName: \"kubernetes.io/projected/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-kube-api-access-rz4rk\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw\" (UID: \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:00.300460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300348 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875s2\" (UniqueName: \"kubernetes.io/projected/c08b4081-5132-4dc8-95b8-df8769d19e57-kube-api-access-875s2\") pod \"maas-api-key-cleanup-29892480-rqwd5\" (UID: \"c08b4081-5132-4dc8-95b8-df8769d19e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:00.300460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300407 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4b893c-251a-445b-b9e7-a1c68571497d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6\" (UID: \"de4b893c-251a-445b-b9e7-a1c68571497d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:00.300460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300451 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08b4081-5132-4dc8-95b8-df8769d19e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892480-rqwd5\" (UID: \"c08b4081-5132-4dc8-95b8-df8769d19e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:00.300611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300508 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchkq\" (UniqueName: \"kubernetes.io/projected/1605fbbc-8000-4cab-a969-b1a6eef41b6d-kube-api-access-gchkq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9\" (UID: \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:00.300611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300551 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3469177a-3810-4974-b737-66199dff79fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r\" (UID: \"3469177a-3810-4974-b737-66199dff79fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:00.300685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300621 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw\" (UID: \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:00.300685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.300647 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldh9\" (UniqueName: \"kubernetes.io/projected/3469177a-3810-4974-b737-66199dff79fd-kube-api-access-jldh9\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r\" (UID: \"3469177a-3810-4974-b737-66199dff79fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:00.301198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.301167 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1605fbbc-8000-4cab-a969-b1a6eef41b6d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9\" (UID: \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:00.301293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.301250 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08b4081-5132-4dc8-95b8-df8769d19e57-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892480-rqwd5\" (UID: \"c08b4081-5132-4dc8-95b8-df8769d19e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:00.301333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.301308 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4b893c-251a-445b-b9e7-a1c68571497d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6\" (UID: \"de4b893c-251a-445b-b9e7-a1c68571497d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:00.301379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.301355 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw\" (UID: \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:00.301413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.301355 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3469177a-3810-4974-b737-66199dff79fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r\" (UID: \"3469177a-3810-4974-b737-66199dff79fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:00.308560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.308522 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm8j5\" (UniqueName: \"kubernetes.io/projected/de4b893c-251a-445b-b9e7-a1c68571497d-kube-api-access-hm8j5\") pod \"maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6\" (UID: \"de4b893c-251a-445b-b9e7-a1c68571497d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:00.308735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.308714 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz4rk\" (UniqueName: \"kubernetes.io/projected/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-kube-api-access-rz4rk\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw\" (UID: \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:00.309067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.309041 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldh9\" (UniqueName: \"kubernetes.io/projected/3469177a-3810-4974-b737-66199dff79fd-kube-api-access-jldh9\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r\" (UID: \"3469177a-3810-4974-b737-66199dff79fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:00.309206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.309184 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchkq\" (UniqueName: \"kubernetes.io/projected/1605fbbc-8000-4cab-a969-b1a6eef41b6d-kube-api-access-gchkq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9\" (UID: \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:00.309312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.309294 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-875s2\" (UniqueName: \"kubernetes.io/projected/c08b4081-5132-4dc8-95b8-df8769d19e57-kube-api-access-875s2\") pod \"maas-api-key-cleanup-29892480-rqwd5\" (UID: \"c08b4081-5132-4dc8-95b8-df8769d19e57\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:00.481021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.480962 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:00.482106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.482047 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:00.491683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.491662 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:00.504250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.504210 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:00.512829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.512801 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:00.580436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.561185 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:00.793212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.793169 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5"] Nov 01 16:00:00.793796 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:00.793768 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc08b4081_5132_4dc8_95b8_df8769d19e57.slice/crio-1c416bfa42fe6d57dc360ae6e45de22a26469060f7717a9ffbef9487e88f6812 WatchSource:0}: Error finding container 1c416bfa42fe6d57dc360ae6e45de22a26469060f7717a9ffbef9487e88f6812: Status 404 returned error can't find the container with id 1c416bfa42fe6d57dc360ae6e45de22a26469060f7717a9ffbef9487e88f6812 Nov 01 16:00:00.795851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.795832 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:00:00.890952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.890924 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9"] Nov 01 16:00:00.893055 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:00.893023 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1605fbbc_8000_4cab_a969_b1a6eef41b6d.slice/crio-3a009fe932d3cf6d08ad122bbd565a12f16d0e7be0f45b01453156edcd27a70a WatchSource:0}: Error finding container 3a009fe932d3cf6d08ad122bbd565a12f16d0e7be0f45b01453156edcd27a70a: Status 404 returned error can't find the container with id 3a009fe932d3cf6d08ad122bbd565a12f16d0e7be0f45b01453156edcd27a70a Nov 01 16:00:00.894334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.894307 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6"] Nov 01 16:00:00.894506 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:00.894487 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde4b893c_251a_445b_b9e7_a1c68571497d.slice/crio-9a1d7b4407b6fd065836572b582aa32b3104923cb90963c726c7b41ccd6974bc WatchSource:0}: Error finding container 9a1d7b4407b6fd065836572b582aa32b3104923cb90963c726c7b41ccd6974bc: Status 404 returned error can't find the container with id 9a1d7b4407b6fd065836572b582aa32b3104923cb90963c726c7b41ccd6974bc Nov 01 16:00:00.914387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:00.914362 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r"] Nov 01 16:00:00.916082 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:00.916054 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3469177a_3810_4974_b737_66199dff79fd.slice/crio-614bff448ff8759369871e0ed92aa447b48a75add86e7bd5e525b74045c3546f WatchSource:0}: Error finding container 614bff448ff8759369871e0ed92aa447b48a75add86e7bd5e525b74045c3546f: Status 404 returned error can't find the container with id 614bff448ff8759369871e0ed92aa447b48a75add86e7bd5e525b74045c3546f Nov 01 16:00:01.009870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.009835 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" event={"ID":"de4b893c-251a-445b-b9e7-a1c68571497d","Type":"ContainerStarted","Data":"9a1d7b4407b6fd065836572b582aa32b3104923cb90963c726c7b41ccd6974bc"} Nov 01 16:00:01.012251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.012209 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" event={"ID":"1605fbbc-8000-4cab-a969-b1a6eef41b6d","Type":"ContainerStarted","Data":"3a009fe932d3cf6d08ad122bbd565a12f16d0e7be0f45b01453156edcd27a70a"} Nov 01 16:00:01.013600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.013574 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" event={"ID":"3469177a-3810-4974-b737-66199dff79fd","Type":"ContainerStarted","Data":"614bff448ff8759369871e0ed92aa447b48a75add86e7bd5e525b74045c3546f"} Nov 01 16:00:01.014995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.014973 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" event={"ID":"c08b4081-5132-4dc8-95b8-df8769d19e57","Type":"ContainerStarted","Data":"1c416bfa42fe6d57dc360ae6e45de22a26469060f7717a9ffbef9487e88f6812"} Nov 01 16:00:01.034440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.034410 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw"] Nov 01 16:00:01.035715 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:01.035690 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e93ce3_bf81_4f68_b659_b96d0b1c0427.slice/crio-948f774383fe44495a1970da31590e1ae335fb0c300806e884a944e6684712db WatchSource:0}: Error finding container 948f774383fe44495a1970da31590e1ae335fb0c300806e884a944e6684712db: Status 404 returned error can't find the container with id 948f774383fe44495a1970da31590e1ae335fb0c300806e884a944e6684712db Nov 01 16:00:01.273602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.273565 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 16:00:01.287316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.286900 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 16:00:01.350564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.350533 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:01.794642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.794609 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl"] Nov 01 16:00:01.801301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.801273 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl"] Nov 01 16:00:01.801466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.801366 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.805178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.805153 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:01.820307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.820279 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621564cb-729f-41dc-986a-865e7928511c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.820445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.820333 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgkrx\" (UniqueName: \"kubernetes.io/projected/621564cb-729f-41dc-986a-865e7928511c-kube-api-access-wgkrx\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.820501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.820453 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621564cb-729f-41dc-986a-865e7928511c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.921624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.921582 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621564cb-729f-41dc-986a-865e7928511c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.921878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.921659 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgkrx\" (UniqueName: \"kubernetes.io/projected/621564cb-729f-41dc-986a-865e7928511c-kube-api-access-wgkrx\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.921878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.921712 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621564cb-729f-41dc-986a-865e7928511c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.922407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.922386 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621564cb-729f-41dc-986a-865e7928511c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.924010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.923985 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621564cb-729f-41dc-986a-865e7928511c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:01.928961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:01.928939 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgkrx\" (UniqueName: \"kubernetes.io/projected/621564cb-729f-41dc-986a-865e7928511c-kube-api-access-wgkrx\") pod \"maas-api-revoke-sub-ea7d7981-jjkfl\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:02.022433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:02.022389 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" event={"ID":"a5e93ce3-bf81-4f68-b659-b96d0b1c0427","Type":"ContainerStarted","Data":"948f774383fe44495a1970da31590e1ae335fb0c300806e884a944e6684712db"} Nov 01 16:00:02.088511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:02.088435 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:02.115741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:02.115700 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:02.282825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:02.278271 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 16:00:02.282825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:02.278341 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 16:00:02.429337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:02.429295 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl"] Nov 01 16:00:02.431006 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:02.430976 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621564cb_729f_41dc_986a_865e7928511c.slice/crio-42302bc9a4ea80de1f213e2e1433ae5563a26864d66dac12d178bc2f00fa555a WatchSource:0}: Error finding container 42302bc9a4ea80de1f213e2e1433ae5563a26864d66dac12d178bc2f00fa555a: Status 404 returned error can't find the container with id 42302bc9a4ea80de1f213e2e1433ae5563a26864d66dac12d178bc2f00fa555a Nov 01 16:00:02.790833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:02.790043 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:03.034591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:03.034505 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" event={"ID":"621564cb-729f-41dc-986a-865e7928511c","Type":"ContainerStarted","Data":"42302bc9a4ea80de1f213e2e1433ae5563a26864d66dac12d178bc2f00fa555a"} Nov 01 16:00:03.288687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:03.288202 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 16:00:03.288687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:03.288271 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 16:00:03.717625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:03.717588 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:04.286795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:04.271235 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 16:00:04.286795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:04.271302 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 16:00:04.443981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:04.443504 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:05.287849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:05.273126 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 16:00:05.287849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:05.273190 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 16:00:05.443399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:05.443363 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:06.267574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:06.267537 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 01 16:00:06.268024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:06.267600 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 01 16:00:06.330097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:06.330033 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:06.965031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:06.964963 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:07.116573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.116532 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" event={"ID":"c08b4081-5132-4dc8-95b8-df8769d19e57","Type":"ContainerStarted","Data":"77c84f6c8aad95e63254e65f90b896262108678c0f3407860c6be61164bc2e48"} Nov 01 16:00:07.118165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.118135 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" event={"ID":"621564cb-729f-41dc-986a-865e7928511c","Type":"ContainerStarted","Data":"09331a4a88221e34c8fb7dd6c24b9cc656a0f70d56f57f6aa74af4700b41529d"} Nov 01 16:00:07.119575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.119550 4573 generic.go:356] "Generic (PLEG): container finished" podID="3469177a-3810-4974-b737-66199dff79fd" containerID="3dd6bf7a756f43a133a8dbc51aedb876957bc49b3665157bd7b182e20ec8ca35" exitCode=60 Nov 01 16:00:07.119656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.119602 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" event={"ID":"3469177a-3810-4974-b737-66199dff79fd","Type":"ContainerDied","Data":"3dd6bf7a756f43a133a8dbc51aedb876957bc49b3665157bd7b182e20ec8ca35"} Nov 01 16:00:07.119812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.119799 4573 scope.go:122] "RemoveContainer" containerID="3dd6bf7a756f43a133a8dbc51aedb876957bc49b3665157bd7b182e20ec8ca35" Nov 01 16:00:07.132468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.132430 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" podStartSLOduration=1.400056081 podStartE2EDuration="7.13241492s" totalImagesPullingTime="5.732358839s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:07.131973143 +0000 UTC m=+744.835139082" watchObservedRunningTime="2026-11-01 16:00:07.13241492 +0000 UTC m=+744.835580856" Nov 01 16:00:07.148764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.148712 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" podStartSLOduration=2.367850747 podStartE2EDuration="6.148694369s" totalImagesPullingTime="3.780843622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:07.146529723 +0000 UTC m=+744.849695669" watchObservedRunningTime="2026-11-01 16:00:07.148694369 +0000 UTC m=+744.851860306" Nov 01 16:00:07.272028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.272004 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 16:00:07.272491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.272466 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 16:00:07.273529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.273507 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 16:00:07.796860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:07.796831 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:08.128054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.128020 4573 generic.go:356] "Generic (PLEG): container finished" podID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerID="a412f02649265e6304ea9e4f84fe4b1cc2d9305412cbe9df554fc06a1c44d998" exitCode=60 Nov 01 16:00:08.128182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.128090 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" event={"ID":"a5e93ce3-bf81-4f68-b659-b96d0b1c0427","Type":"ContainerDied","Data":"a412f02649265e6304ea9e4f84fe4b1cc2d9305412cbe9df554fc06a1c44d998"} Nov 01 16:00:08.128428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.128412 4573 scope.go:122] "RemoveContainer" containerID="a412f02649265e6304ea9e4f84fe4b1cc2d9305412cbe9df554fc06a1c44d998" Nov 01 16:00:08.129797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.129774 4573 generic.go:356] "Generic (PLEG): container finished" podID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerID="77c84f6c8aad95e63254e65f90b896262108678c0f3407860c6be61164bc2e48" exitCode=60 Nov 01 16:00:08.129905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.129805 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" event={"ID":"c08b4081-5132-4dc8-95b8-df8769d19e57","Type":"ContainerDied","Data":"77c84f6c8aad95e63254e65f90b896262108678c0f3407860c6be61164bc2e48"} Nov 01 16:00:08.130175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.130114 4573 scope.go:122] "RemoveContainer" containerID="77c84f6c8aad95e63254e65f90b896262108678c0f3407860c6be61164bc2e48" Nov 01 16:00:08.132432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.132411 4573 generic.go:356] "Generic (PLEG): container finished" podID="621564cb-729f-41dc-986a-865e7928511c" containerID="09331a4a88221e34c8fb7dd6c24b9cc656a0f70d56f57f6aa74af4700b41529d" exitCode=0 Nov 01 16:00:08.132543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.132508 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" event={"ID":"621564cb-729f-41dc-986a-865e7928511c","Type":"ContainerDied","Data":"09331a4a88221e34c8fb7dd6c24b9cc656a0f70d56f57f6aa74af4700b41529d"} Nov 01 16:00:08.134097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.134076 4573 generic.go:356] "Generic (PLEG): container finished" podID="de4b893c-251a-445b-b9e7-a1c68571497d" containerID="d767de4f4b212aed4ab08f3b60d7ab7b7d7aaf870d4319eab939a6ed61c30a06" exitCode=60 Nov 01 16:00:08.134214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.134138 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" event={"ID":"de4b893c-251a-445b-b9e7-a1c68571497d","Type":"ContainerDied","Data":"d767de4f4b212aed4ab08f3b60d7ab7b7d7aaf870d4319eab939a6ed61c30a06"} Nov 01 16:00:08.134405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.134366 4573 scope.go:122] "RemoveContainer" containerID="d767de4f4b212aed4ab08f3b60d7ab7b7d7aaf870d4319eab939a6ed61c30a06" Nov 01 16:00:08.135618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.135596 4573 generic.go:356] "Generic (PLEG): container finished" podID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerID="4cb63cc16c46b28b455535ab19ce7e23efed944ac38ff8f2d528f1ef5bea3f9a" exitCode=60 Nov 01 16:00:08.135686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.135620 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" event={"ID":"1605fbbc-8000-4cab-a969-b1a6eef41b6d","Type":"ContainerDied","Data":"4cb63cc16c46b28b455535ab19ce7e23efed944ac38ff8f2d528f1ef5bea3f9a"} Nov 01 16:00:08.136393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.136374 4573 scope.go:122] "RemoveContainer" containerID="4cb63cc16c46b28b455535ab19ce7e23efed944ac38ff8f2d528f1ef5bea3f9a" Nov 01 16:00:08.137544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.137525 4573 generic.go:356] "Generic (PLEG): container finished" podID="3469177a-3810-4974-b737-66199dff79fd" containerID="ebe815b27321efd38a080b2830495d436c7ddca693195ab08523799dbf4ec289" exitCode=60 Nov 01 16:00:08.137625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.137576 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" event={"ID":"3469177a-3810-4974-b737-66199dff79fd","Type":"ContainerDied","Data":"ebe815b27321efd38a080b2830495d436c7ddca693195ab08523799dbf4ec289"} Nov 01 16:00:08.137625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.137618 4573 scope.go:122] "RemoveContainer" containerID="3dd6bf7a756f43a133a8dbc51aedb876957bc49b3665157bd7b182e20ec8ca35" Nov 01 16:00:08.137829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.137807 4573 scope.go:122] "RemoveContainer" containerID="ebe815b27321efd38a080b2830495d436c7ddca693195ab08523799dbf4ec289" Nov 01 16:00:08.138028 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:08.138009 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r_odh-ai-gateway-infra(3469177a-3810-4974-b737-66199dff79fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" podUID="3469177a-3810-4974-b737-66199dff79fd" Nov 01 16:00:08.777525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.777497 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:08.810212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.810180 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s"] Nov 01 16:00:08.814399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.814384 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:08.822684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.822645 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s"] Nov 01 16:00:08.968355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.968263 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvsr\" (UniqueName: \"kubernetes.io/projected/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-kube-api-access-6wvsr\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:08.968355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.968309 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:08.968546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:08.968388 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:09.070133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.070097 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvsr\" (UniqueName: \"kubernetes.io/projected/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-kube-api-access-6wvsr\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:09.070296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.070158 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:09.070296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.070195 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:09.071537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.070840 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:09.072724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.072702 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:09.077446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.077424 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvsr\" (UniqueName: \"kubernetes.io/projected/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-kube-api-access-6wvsr\") pod \"maas-api-revoke-sub-fdd08a9d-s7z4s\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:09.129919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.129887 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:09.149224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.148722 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" event={"ID":"c08b4081-5132-4dc8-95b8-df8769d19e57","Type":"ContainerStarted","Data":"0788d5ccb3432602d3f3b9d9874a2a92a11164739bc85769517c1e707997326f"} Nov 01 16:00:09.151795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.151713 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" event={"ID":"de4b893c-251a-445b-b9e7-a1c68571497d","Type":"ContainerStarted","Data":"2303d86a781c2bddb46c8ddc94bf1a24c655f81ce4d78295d355ff80bab4a05d"} Nov 01 16:00:09.154317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.154290 4573 generic.go:356] "Generic (PLEG): container finished" podID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerID="9f8ba143f535cac6630dca1942da3c976198c98b940e1450d03eace92a1229ef" exitCode=60 Nov 01 16:00:09.154427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.154364 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" event={"ID":"1605fbbc-8000-4cab-a969-b1a6eef41b6d","Type":"ContainerDied","Data":"9f8ba143f535cac6630dca1942da3c976198c98b940e1450d03eace92a1229ef"} Nov 01 16:00:09.154427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.154420 4573 scope.go:122] "RemoveContainer" containerID="4cb63cc16c46b28b455535ab19ce7e23efed944ac38ff8f2d528f1ef5bea3f9a" Nov 01 16:00:09.155294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.155161 4573 scope.go:122] "RemoveContainer" containerID="9f8ba143f535cac6630dca1942da3c976198c98b940e1450d03eace92a1229ef" Nov 01 16:00:09.155711 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:09.155483 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9_odh-ai-gateway-infra(1605fbbc-8000-4cab-a969-b1a6eef41b6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" Nov 01 16:00:09.157022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.156965 4573 scope.go:122] "RemoveContainer" containerID="ebe815b27321efd38a080b2830495d436c7ddca693195ab08523799dbf4ec289" Nov 01 16:00:09.157575 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:09.157467 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r_odh-ai-gateway-infra(3469177a-3810-4974-b737-66199dff79fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" podUID="3469177a-3810-4974-b737-66199dff79fd" Nov 01 16:00:09.160409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.159744 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" event={"ID":"a5e93ce3-bf81-4f68-b659-b96d0b1c0427","Type":"ContainerStarted","Data":"7c50603d789456ae7ec782ed58f9acc900587e3217bb8405fcd0977a79becd8a"} Nov 01 16:00:09.214602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.214538 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" podStartSLOduration=3.041383703 podStartE2EDuration="9.214512623s" totalImagesPullingTime="6.17312892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:09.203440219 +0000 UTC m=+746.906606161" watchObservedRunningTime="2026-11-01 16:00:09.214512623 +0000 UTC m=+746.917678753" Nov 01 16:00:09.428442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.428390 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" podStartSLOduration=3.390970757 podStartE2EDuration="9.428370953s" totalImagesPullingTime="6.037400196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:09.241917201 +0000 UTC m=+746.945083137" watchObservedRunningTime="2026-11-01 16:00:09.428370953 +0000 UTC m=+747.131536890" Nov 01 16:00:09.428985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.428967 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:09.429343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.429322 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s"] Nov 01 16:00:09.430111 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:09.430092 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21b1b3f3_59a4_41b3_8225_7c6e2ed39fca.slice/crio-a64ee3741e846d25e27f483b9d11849edfa060411016af687c522e2a9c5635a7 WatchSource:0}: Error finding container a64ee3741e846d25e27f483b9d11849edfa060411016af687c522e2a9c5635a7: Status 404 returned error can't find the container with id a64ee3741e846d25e27f483b9d11849edfa060411016af687c522e2a9c5635a7 Nov 01 16:00:09.485766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.485723 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:09.578913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.578818 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/621564cb-729f-41dc-986a-865e7928511c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621564cb-729f-41dc-986a-865e7928511c-maas-api-xks-ca\") pod \"621564cb-729f-41dc-986a-865e7928511c\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " Nov 01 16:00:09.579062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.578917 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/621564cb-729f-41dc-986a-865e7928511c-kube-api-access-wgkrx\" (UniqueName: \"kubernetes.io/projected/621564cb-729f-41dc-986a-865e7928511c-kube-api-access-wgkrx\") pod \"621564cb-729f-41dc-986a-865e7928511c\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " Nov 01 16:00:09.579062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.579027 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/621564cb-729f-41dc-986a-865e7928511c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621564cb-729f-41dc-986a-865e7928511c-maas-api-openshift-ca\") pod \"621564cb-729f-41dc-986a-865e7928511c\" (UID: \"621564cb-729f-41dc-986a-865e7928511c\") " Nov 01 16:00:09.579421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.579382 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/621564cb-729f-41dc-986a-865e7928511c-maas-api-openshift-ca" pod "621564cb-729f-41dc-986a-865e7928511c" (UID: "621564cb-729f-41dc-986a-865e7928511c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:09.580615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.579895 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/621564cb-729f-41dc-986a-865e7928511c-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:09.581512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.581487 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/621564cb-729f-41dc-986a-865e7928511c-maas-api-xks-ca" pod "621564cb-729f-41dc-986a-865e7928511c" (UID: "621564cb-729f-41dc-986a-865e7928511c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:09.581632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.581612 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/621564cb-729f-41dc-986a-865e7928511c-kube-api-access-wgkrx" pod "621564cb-729f-41dc-986a-865e7928511c" (UID: "621564cb-729f-41dc-986a-865e7928511c"). InnerVolumeSpecName "kube-api-access-wgkrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:09.680604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.680569 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/621564cb-729f-41dc-986a-865e7928511c-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:09.680604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:09.680597 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgkrx\" (UniqueName: \"kubernetes.io/projected/621564cb-729f-41dc-986a-865e7928511c-kube-api-access-wgkrx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:10.168259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.168180 4573 generic.go:356] "Generic (PLEG): container finished" podID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerID="0788d5ccb3432602d3f3b9d9874a2a92a11164739bc85769517c1e707997326f" exitCode=60 Nov 01 16:00:10.168259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.168235 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" event={"ID":"c08b4081-5132-4dc8-95b8-df8769d19e57","Type":"ContainerDied","Data":"0788d5ccb3432602d3f3b9d9874a2a92a11164739bc85769517c1e707997326f"} Nov 01 16:00:10.168925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.168295 4573 scope.go:122] "RemoveContainer" containerID="77c84f6c8aad95e63254e65f90b896262108678c0f3407860c6be61164bc2e48" Nov 01 16:00:10.168925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.168571 4573 scope.go:122] "RemoveContainer" containerID="0788d5ccb3432602d3f3b9d9874a2a92a11164739bc85769517c1e707997326f" Nov 01 16:00:10.168925 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:10.168901 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892480-rqwd5_odh-ai-gateway-infra(c08b4081-5132-4dc8-95b8-df8769d19e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" Nov 01 16:00:10.169949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.169930 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" Nov 01 16:00:10.170131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.169965 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl" event={"ID":"621564cb-729f-41dc-986a-865e7928511c","Type":"ContainerDied","Data":"42302bc9a4ea80de1f213e2e1433ae5563a26864d66dac12d178bc2f00fa555a"} Nov 01 16:00:10.170131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.170004 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42302bc9a4ea80de1f213e2e1433ae5563a26864d66dac12d178bc2f00fa555a" Nov 01 16:00:10.172624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.172602 4573 generic.go:356] "Generic (PLEG): container finished" podID="de4b893c-251a-445b-b9e7-a1c68571497d" containerID="2303d86a781c2bddb46c8ddc94bf1a24c655f81ce4d78295d355ff80bab4a05d" exitCode=60 Nov 01 16:00:10.172759 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.172679 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" event={"ID":"de4b893c-251a-445b-b9e7-a1c68571497d","Type":"ContainerDied","Data":"2303d86a781c2bddb46c8ddc94bf1a24c655f81ce4d78295d355ff80bab4a05d"} Nov 01 16:00:10.173002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.172974 4573 scope.go:122] "RemoveContainer" containerID="2303d86a781c2bddb46c8ddc94bf1a24c655f81ce4d78295d355ff80bab4a05d" Nov 01 16:00:10.173199 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:10.173180 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6_odh-ai-gateway-infra(de4b893c-251a-445b-b9e7-a1c68571497d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" Nov 01 16:00:10.175311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.175292 4573 scope.go:122] "RemoveContainer" containerID="9f8ba143f535cac6630dca1942da3c976198c98b940e1450d03eace92a1229ef" Nov 01 16:00:10.175534 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:10.175507 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9_odh-ai-gateway-infra(1605fbbc-8000-4cab-a969-b1a6eef41b6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" Nov 01 16:00:10.177016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.176995 4573 generic.go:356] "Generic (PLEG): container finished" podID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerID="7c50603d789456ae7ec782ed58f9acc900587e3217bb8405fcd0977a79becd8a" exitCode=60 Nov 01 16:00:10.177105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.177061 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" event={"ID":"a5e93ce3-bf81-4f68-b659-b96d0b1c0427","Type":"ContainerDied","Data":"7c50603d789456ae7ec782ed58f9acc900587e3217bb8405fcd0977a79becd8a"} Nov 01 16:00:10.177276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.177263 4573 scope.go:122] "RemoveContainer" containerID="7c50603d789456ae7ec782ed58f9acc900587e3217bb8405fcd0977a79becd8a" Nov 01 16:00:10.177448 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:10.177432 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw_odh-ai-gateway-infra(a5e93ce3-bf81-4f68-b659-b96d0b1c0427)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" Nov 01 16:00:10.179170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.179146 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" event={"ID":"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca","Type":"ContainerStarted","Data":"13f8300f3806945534375fe4a3cc3b6647802a565a83c69e01976a34e48dc813"} Nov 01 16:00:10.179252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.179195 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" event={"ID":"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca","Type":"ContainerStarted","Data":"a64ee3741e846d25e27f483b9d11849edfa060411016af687c522e2a9c5635a7"} Nov 01 16:00:10.187818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.187793 4573 scope.go:122] "RemoveContainer" containerID="d767de4f4b212aed4ab08f3b60d7ab7b7d7aaf870d4319eab939a6ed61c30a06" Nov 01 16:00:10.195091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.195068 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:10.200715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.200697 4573 scope.go:122] "RemoveContainer" containerID="a412f02649265e6304ea9e4f84fe4b1cc2d9305412cbe9df554fc06a1c44d998" Nov 01 16:00:10.259403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:10.259358 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" podStartSLOduration=2.259344497 podStartE2EDuration="2.259344497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:10.258891507 +0000 UTC m=+747.962057443" watchObservedRunningTime="2026-11-01 16:00:10.259344497 +0000 UTC m=+747.962510433" Nov 01 16:00:11.208794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:11.207799 4573 scope.go:122] "RemoveContainer" containerID="2303d86a781c2bddb46c8ddc94bf1a24c655f81ce4d78295d355ff80bab4a05d" Nov 01 16:00:11.208794 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:11.208153 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6_odh-ai-gateway-infra(de4b893c-251a-445b-b9e7-a1c68571497d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" Nov 01 16:00:11.226504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:11.225791 4573 scope.go:122] "RemoveContainer" containerID="7c50603d789456ae7ec782ed58f9acc900587e3217bb8405fcd0977a79becd8a" Nov 01 16:00:11.226504 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:11.226121 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw_odh-ai-gateway-infra(a5e93ce3-bf81-4f68-b659-b96d0b1c0427)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" Nov 01 16:00:11.237801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:11.237094 4573 generic.go:356] "Generic (PLEG): container finished" podID="21b1b3f3-59a4-41b3-8225-7c6e2ed39fca" containerID="13f8300f3806945534375fe4a3cc3b6647802a565a83c69e01976a34e48dc813" exitCode=0 Nov 01 16:00:11.237801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:11.237244 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" event={"ID":"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca","Type":"ContainerDied","Data":"13f8300f3806945534375fe4a3cc3b6647802a565a83c69e01976a34e48dc813"} Nov 01 16:00:11.241346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:11.240640 4573 scope.go:122] "RemoveContainer" containerID="0788d5ccb3432602d3f3b9d9874a2a92a11164739bc85769517c1e707997326f" Nov 01 16:00:11.241346 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:11.240935 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892480-rqwd5_odh-ai-gateway-infra(c08b4081-5132-4dc8-95b8-df8769d19e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" Nov 01 16:00:11.367593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:11.367555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:12.152239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.152202 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:12.390674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.390644 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:12.540681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.540589 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-xks-ca\") pod \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " Nov 01 16:00:12.540873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.540717 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-kube-api-access-6wvsr\" (UniqueName: \"kubernetes.io/projected/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-kube-api-access-6wvsr\") pod \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " Nov 01 16:00:12.540873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.540857 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-openshift-ca\") pod \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\" (UID: \"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca\") " Nov 01 16:00:12.541421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.541182 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-openshift-ca" pod "21b1b3f3-59a4-41b3-8225-7c6e2ed39fca" (UID: "21b1b3f3-59a4-41b3-8225-7c6e2ed39fca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:12.541543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.541466 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:12.543005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.542974 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-kube-api-access-6wvsr" pod "21b1b3f3-59a4-41b3-8225-7c6e2ed39fca" (UID: "21b1b3f3-59a4-41b3-8225-7c6e2ed39fca"). InnerVolumeSpecName "kube-api-access-6wvsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:12.543101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.543040 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-xks-ca" pod "21b1b3f3-59a4-41b3-8225-7c6e2ed39fca" (UID: "21b1b3f3-59a4-41b3-8225-7c6e2ed39fca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:12.642825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.642795 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:12.642825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.642825 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wvsr\" (UniqueName: \"kubernetes.io/projected/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca-kube-api-access-6wvsr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:12.862330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:12.862293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:13.254378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:13.254295 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" event={"ID":"21b1b3f3-59a4-41b3-8225-7c6e2ed39fca","Type":"ContainerDied","Data":"a64ee3741e846d25e27f483b9d11849edfa060411016af687c522e2a9c5635a7"} Nov 01 16:00:13.254378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:13.254344 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a64ee3741e846d25e27f483b9d11849edfa060411016af687c522e2a9c5635a7" Nov 01 16:00:13.254378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:13.254301 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s" Nov 01 16:00:13.667006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:13.666971 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:14.374376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:14.374340 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:15.066718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.066682 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:15.695060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.695027 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q"] Nov 01 16:00:15.695576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.695563 4573 cpu_manager.go:414] "containerMap: removing container" podUID="621564cb-729f-41dc-986a-865e7928511c" containerName="revoke-keys" Nov 01 16:00:15.695620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.695579 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="621564cb-729f-41dc-986a-865e7928511c" containerName="revoke-keys" Nov 01 16:00:15.695620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.695615 4573 cpu_manager.go:414] "containerMap: removing container" podUID="21b1b3f3-59a4-41b3-8225-7c6e2ed39fca" containerName="revoke-keys" Nov 01 16:00:15.695686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.695620 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21b1b3f3-59a4-41b3-8225-7c6e2ed39fca" containerName="revoke-keys" Nov 01 16:00:15.695719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.695696 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="21b1b3f3-59a4-41b3-8225-7c6e2ed39fca" containerName="revoke-keys" Nov 01 16:00:15.695719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.695710 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="621564cb-729f-41dc-986a-865e7928511c" containerName="revoke-keys" Nov 01 16:00:15.700589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.700572 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.704999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.703680 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:15.704999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.704941 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q"] Nov 01 16:00:15.758817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.758790 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:15.775254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.775224 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.775383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.775288 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w275p\" (UniqueName: \"kubernetes.io/projected/3c48ac04-6dda-407b-9803-4875988372ca-kube-api-access-w275p\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.775441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.775421 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.876527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.876493 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w275p\" (UniqueName: \"kubernetes.io/projected/3c48ac04-6dda-407b-9803-4875988372ca-kube-api-access-w275p\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.876701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.876592 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.876701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.876675 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.877356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.877332 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.878896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.878876 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:15.884177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:15.884152 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w275p\" (UniqueName: \"kubernetes.io/projected/3c48ac04-6dda-407b-9803-4875988372ca-kube-api-access-w275p\") pod \"maas-api-revoke-sub-61f137f8-t7b4q\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:16.019948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:16.019872 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:16.300924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:16.300895 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q"] Nov 01 16:00:16.302420 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:16.302395 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c48ac04_6dda_407b_9803_4875988372ca.slice/crio-77cf7b6d0905acc5478af4fc9dba7286d0af7f9479411355b1c1a524474aeb67 WatchSource:0}: Error finding container 77cf7b6d0905acc5478af4fc9dba7286d0af7f9479411355b1c1a524474aeb67: Status 404 returned error can't find the container with id 77cf7b6d0905acc5478af4fc9dba7286d0af7f9479411355b1c1a524474aeb67 Nov 01 16:00:16.582180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:16.582151 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:17.280551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:17.280517 4573 generic.go:356] "Generic (PLEG): container finished" podID="3c48ac04-6dda-407b-9803-4875988372ca" containerID="5a7818597638b23364abac4a396d24e1a5dbc12d6123c5be7c01d3401fe7c53e" exitCode=0 Nov 01 16:00:17.280737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:17.280603 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" event={"ID":"3c48ac04-6dda-407b-9803-4875988372ca","Type":"ContainerDied","Data":"5a7818597638b23364abac4a396d24e1a5dbc12d6123c5be7c01d3401fe7c53e"} Nov 01 16:00:17.280737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:17.280651 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" event={"ID":"3c48ac04-6dda-407b-9803-4875988372ca","Type":"ContainerStarted","Data":"77cf7b6d0905acc5478af4fc9dba7286d0af7f9479411355b1c1a524474aeb67"} Nov 01 16:00:17.458637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:17.458573 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:18.360842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.360639 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:18.471770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.471731 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:18.507511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.507481 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-xks-ca\") pod \"3c48ac04-6dda-407b-9803-4875988372ca\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " Nov 01 16:00:18.507667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.507523 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c48ac04-6dda-407b-9803-4875988372ca-kube-api-access-w275p\" (UniqueName: \"kubernetes.io/projected/3c48ac04-6dda-407b-9803-4875988372ca-kube-api-access-w275p\") pod \"3c48ac04-6dda-407b-9803-4875988372ca\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " Nov 01 16:00:18.507667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.507596 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-openshift-ca\") pod \"3c48ac04-6dda-407b-9803-4875988372ca\" (UID: \"3c48ac04-6dda-407b-9803-4875988372ca\") " Nov 01 16:00:18.508047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.508020 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-openshift-ca" pod "3c48ac04-6dda-407b-9803-4875988372ca" (UID: "3c48ac04-6dda-407b-9803-4875988372ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:18.510193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.510168 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-xks-ca" pod "3c48ac04-6dda-407b-9803-4875988372ca" (UID: "3c48ac04-6dda-407b-9803-4875988372ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:18.510319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.510205 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c48ac04-6dda-407b-9803-4875988372ca-kube-api-access-w275p" pod "3c48ac04-6dda-407b-9803-4875988372ca" (UID: "3c48ac04-6dda-407b-9803-4875988372ca"). InnerVolumeSpecName "kube-api-access-w275p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:18.609141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.609114 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:18.609292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.609144 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c48ac04-6dda-407b-9803-4875988372ca-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:18.609292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:18.609159 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w275p\" (UniqueName: \"kubernetes.io/projected/3c48ac04-6dda-407b-9803-4875988372ca-kube-api-access-w275p\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:19.051091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:19.051005 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:19.294240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:19.294211 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" Nov 01 16:00:19.294240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:19.294222 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q" event={"ID":"3c48ac04-6dda-407b-9803-4875988372ca","Type":"ContainerDied","Data":"77cf7b6d0905acc5478af4fc9dba7286d0af7f9479411355b1c1a524474aeb67"} Nov 01 16:00:19.294452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:19.294289 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77cf7b6d0905acc5478af4fc9dba7286d0af7f9479411355b1c1a524474aeb67" Nov 01 16:00:19.883426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:19.883388 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:20.389139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.389111 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f"] Nov 01 16:00:20.389694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.389681 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3c48ac04-6dda-407b-9803-4875988372ca" containerName="revoke-keys" Nov 01 16:00:20.389738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.389695 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c48ac04-6dda-407b-9803-4875988372ca" containerName="revoke-keys" Nov 01 16:00:20.389838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.389820 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c48ac04-6dda-407b-9803-4875988372ca" containerName="revoke-keys" Nov 01 16:00:20.393385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.393366 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.397915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.397890 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:20.401505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.401481 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f"] Nov 01 16:00:20.429693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.429654 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj554\" (UniqueName: \"kubernetes.io/projected/287fbc1d-571f-41ae-afb9-251f2da0d568-kube-api-access-lj554\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.429876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.429722 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.429876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.429853 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.458133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.458113 4573 scope.go:122] "RemoveContainer" containerID="ebe815b27321efd38a080b2830495d436c7ddca693195ab08523799dbf4ec289" Nov 01 16:00:20.531125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.531098 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj554\" (UniqueName: \"kubernetes.io/projected/287fbc1d-571f-41ae-afb9-251f2da0d568-kube-api-access-lj554\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.531249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.531149 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.531334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.531314 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.531914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.531895 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.533179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.533162 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.540066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.540043 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj554\" (UniqueName: \"kubernetes.io/projected/287fbc1d-571f-41ae-afb9-251f2da0d568-kube-api-access-lj554\") pod \"maas-api-revoke-sub-9472f900-zn84f\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.668646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.668579 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:20.709266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.709239 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:20.990886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:20.990856 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f"] Nov 01 16:00:20.992359 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:20.992333 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod287fbc1d_571f_41ae_afb9_251f2da0d568.slice/crio-29a444b1a9063c321911d446833bd529ca2e9e7df0c4ed74bf12461ac188dc85 WatchSource:0}: Error finding container 29a444b1a9063c321911d446833bd529ca2e9e7df0c4ed74bf12461ac188dc85: Status 404 returned error can't find the container with id 29a444b1a9063c321911d446833bd529ca2e9e7df0c4ed74bf12461ac188dc85 Nov 01 16:00:21.307811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.307783 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" event={"ID":"287fbc1d-571f-41ae-afb9-251f2da0d568","Type":"ContainerStarted","Data":"b395e4f898e39520352af0d72ac54efff19e89b0aa98bdc064516f5e2f7446f1"} Nov 01 16:00:21.307932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.307834 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" event={"ID":"287fbc1d-571f-41ae-afb9-251f2da0d568","Type":"ContainerStarted","Data":"29a444b1a9063c321911d446833bd529ca2e9e7df0c4ed74bf12461ac188dc85"} Nov 01 16:00:21.309639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.309619 4573 generic.go:356] "Generic (PLEG): container finished" podID="3469177a-3810-4974-b737-66199dff79fd" containerID="673266b8ddc537be125b9fa514e177d0bd9b94cc1a46006ea5bfbfee0e51f0fe" exitCode=60 Nov 01 16:00:21.309723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.309694 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" event={"ID":"3469177a-3810-4974-b737-66199dff79fd","Type":"ContainerDied","Data":"673266b8ddc537be125b9fa514e177d0bd9b94cc1a46006ea5bfbfee0e51f0fe"} Nov 01 16:00:21.309788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.309744 4573 scope.go:122] "RemoveContainer" containerID="ebe815b27321efd38a080b2830495d436c7ddca693195ab08523799dbf4ec289" Nov 01 16:00:21.310066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.310052 4573 scope.go:122] "RemoveContainer" containerID="673266b8ddc537be125b9fa514e177d0bd9b94cc1a46006ea5bfbfee0e51f0fe" Nov 01 16:00:21.310324 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:21.310303 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r_odh-ai-gateway-infra(3469177a-3810-4974-b737-66199dff79fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" podUID="3469177a-3810-4974-b737-66199dff79fd" Nov 01 16:00:21.341854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.341806 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" podStartSLOduration=1.34179024 podStartE2EDuration="1.34179024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:21.338173985 +0000 UTC m=+759.041339921" watchObservedRunningTime="2026-11-01 16:00:21.34179024 +0000 UTC m=+759.044956179" Nov 01 16:00:21.450723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.450689 4573 scope.go:122] "RemoveContainer" containerID="9f8ba143f535cac6630dca1942da3c976198c98b940e1450d03eace92a1229ef" Nov 01 16:00:21.450932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.450915 4573 scope.go:122] "RemoveContainer" containerID="2303d86a781c2bddb46c8ddc94bf1a24c655f81ce4d78295d355ff80bab4a05d" Nov 01 16:00:21.597629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:21.597593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:22.320614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.320528 4573 generic.go:356] "Generic (PLEG): container finished" podID="287fbc1d-571f-41ae-afb9-251f2da0d568" containerID="b395e4f898e39520352af0d72ac54efff19e89b0aa98bdc064516f5e2f7446f1" exitCode=0 Nov 01 16:00:22.321088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.320640 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" event={"ID":"287fbc1d-571f-41ae-afb9-251f2da0d568","Type":"ContainerDied","Data":"b395e4f898e39520352af0d72ac54efff19e89b0aa98bdc064516f5e2f7446f1"} Nov 01 16:00:22.323986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.323744 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" event={"ID":"de4b893c-251a-445b-b9e7-a1c68571497d","Type":"ContainerStarted","Data":"61bf9540bd97e1758a78fa1c89e3b901f27c590a4341ce610ae0efc412e1fbae"} Nov 01 16:00:22.326651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.326625 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" event={"ID":"1605fbbc-8000-4cab-a969-b1a6eef41b6d","Type":"ContainerStarted","Data":"7728e4083f4a8ea79bed40fcb63cb3089d5e6c92f3f6647f3cd3ab1f9cf9c064"} Nov 01 16:00:22.380373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.380339 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r"] Nov 01 16:00:22.390986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.390927 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" podStartSLOduration=16.512140499 podStartE2EDuration="22.390907558s" totalImagesPullingTime="5.878767059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:22.38631252 +0000 UTC m=+760.089478459" watchObservedRunningTime="2026-11-01 16:00:22.390907558 +0000 UTC m=+760.094073495" Nov 01 16:00:22.396915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.396885 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d"] Nov 01 16:00:22.414441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.414206 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d"] Nov 01 16:00:22.414441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.414319 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.456015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.455981 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.456335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.456311 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.456476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.456459 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvxb\" (UniqueName: \"kubernetes.io/projected/6ab8c178-c15d-4009-a49d-611014f91f05-kube-api-access-lqvxb\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.456988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.456104 4573 scope.go:122] "RemoveContainer" containerID="7c50603d789456ae7ec782ed58f9acc900587e3217bb8405fcd0977a79becd8a" Nov 01 16:00:22.480143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.480119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:22.558798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.558213 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.558798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.558364 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.558798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.558400 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvxb\" (UniqueName: \"kubernetes.io/projected/6ab8c178-c15d-4009-a49d-611014f91f05-kube-api-access-lqvxb\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.559552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.559157 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.560946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.560911 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.570038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.570000 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvxb\" (UniqueName: \"kubernetes.io/projected/6ab8c178-c15d-4009-a49d-611014f91f05-kube-api-access-lqvxb\") pod \"maas-api-revoke-sub-a862ace3-5c94d\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.615416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.615385 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:22.659635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.659607 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3469177a-3810-4974-b737-66199dff79fd-kube-api-access-jldh9\" (UniqueName: \"kubernetes.io/projected/3469177a-3810-4974-b737-66199dff79fd-kube-api-access-jldh9\") pod \"3469177a-3810-4974-b737-66199dff79fd\" (UID: \"3469177a-3810-4974-b737-66199dff79fd\") " Nov 01 16:00:22.659839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.659667 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3469177a-3810-4974-b737-66199dff79fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3469177a-3810-4974-b737-66199dff79fd-maas-api-service-ca\") pod \"3469177a-3810-4974-b737-66199dff79fd\" (UID: \"3469177a-3810-4974-b737-66199dff79fd\") " Nov 01 16:00:22.660129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.660098 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3469177a-3810-4974-b737-66199dff79fd-maas-api-service-ca" pod "3469177a-3810-4974-b737-66199dff79fd" (UID: "3469177a-3810-4974-b737-66199dff79fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:22.661891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.661862 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3469177a-3810-4974-b737-66199dff79fd-kube-api-access-jldh9" pod "3469177a-3810-4974-b737-66199dff79fd" (UID: "3469177a-3810-4974-b737-66199dff79fd"). InnerVolumeSpecName "kube-api-access-jldh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:22.740796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.740764 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:22.761600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.761573 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jldh9\" (UniqueName: \"kubernetes.io/projected/3469177a-3810-4974-b737-66199dff79fd-kube-api-access-jldh9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:22.761600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:22.761598 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3469177a-3810-4974-b737-66199dff79fd-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.027038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.027013 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d"] Nov 01 16:00:23.027984 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:23.027957 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab8c178_c15d_4009_a49d_611014f91f05.slice/crio-3803d199676b3e692ef5ff46cc1c2064c706838fe8283db2ffced33cc00127f2 WatchSource:0}: Error finding container 3803d199676b3e692ef5ff46cc1c2064c706838fe8283db2ffced33cc00127f2: Status 404 returned error can't find the container with id 3803d199676b3e692ef5ff46cc1c2064c706838fe8283db2ffced33cc00127f2 Nov 01 16:00:23.262548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.262461 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:23.355682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.355641 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" event={"ID":"a5e93ce3-bf81-4f68-b659-b96d0b1c0427","Type":"ContainerStarted","Data":"931dbdcd071197b8ca0f6bb468ebe37edb3bb08832968dba6897db53cad125e5"} Nov 01 16:00:23.357035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.357009 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" event={"ID":"6ab8c178-c15d-4009-a49d-611014f91f05","Type":"ContainerStarted","Data":"3803d199676b3e692ef5ff46cc1c2064c706838fe8283db2ffced33cc00127f2"} Nov 01 16:00:23.358758 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.358725 4573 generic.go:356] "Generic (PLEG): container finished" podID="de4b893c-251a-445b-b9e7-a1c68571497d" containerID="61bf9540bd97e1758a78fa1c89e3b901f27c590a4341ce610ae0efc412e1fbae" exitCode=60 Nov 01 16:00:23.358855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.358776 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" event={"ID":"de4b893c-251a-445b-b9e7-a1c68571497d","Type":"ContainerDied","Data":"61bf9540bd97e1758a78fa1c89e3b901f27c590a4341ce610ae0efc412e1fbae"} Nov 01 16:00:23.358855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.358827 4573 scope.go:122] "RemoveContainer" containerID="2303d86a781c2bddb46c8ddc94bf1a24c655f81ce4d78295d355ff80bab4a05d" Nov 01 16:00:23.359131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.359117 4573 scope.go:122] "RemoveContainer" containerID="61bf9540bd97e1758a78fa1c89e3b901f27c590a4341ce610ae0efc412e1fbae" Nov 01 16:00:23.359429 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:23.359412 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6_odh-ai-gateway-infra(de4b893c-251a-445b-b9e7-a1c68571497d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" Nov 01 16:00:23.361235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.361217 4573 generic.go:356] "Generic (PLEG): container finished" podID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerID="7728e4083f4a8ea79bed40fcb63cb3089d5e6c92f3f6647f3cd3ab1f9cf9c064" exitCode=60 Nov 01 16:00:23.361322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.361284 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" event={"ID":"1605fbbc-8000-4cab-a969-b1a6eef41b6d","Type":"ContainerDied","Data":"7728e4083f4a8ea79bed40fcb63cb3089d5e6c92f3f6647f3cd3ab1f9cf9c064"} Nov 01 16:00:23.361597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.361579 4573 scope.go:122] "RemoveContainer" containerID="7728e4083f4a8ea79bed40fcb63cb3089d5e6c92f3f6647f3cd3ab1f9cf9c064" Nov 01 16:00:23.361870 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:23.361846 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9_odh-ai-gateway-infra(1605fbbc-8000-4cab-a969-b1a6eef41b6d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" Nov 01 16:00:23.362557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.362543 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" Nov 01 16:00:23.362643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.362581 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r" event={"ID":"3469177a-3810-4974-b737-66199dff79fd","Type":"ContainerDied","Data":"614bff448ff8759369871e0ed92aa447b48a75add86e7bd5e525b74045c3546f"} Nov 01 16:00:23.374721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.374675 4573 scope.go:122] "RemoveContainer" containerID="9f8ba143f535cac6630dca1942da3c976198c98b940e1450d03eace92a1229ef" Nov 01 16:00:23.375272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.375254 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6"] Nov 01 16:00:23.388373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.388360 4573 scope.go:122] "RemoveContainer" containerID="673266b8ddc537be125b9fa514e177d0bd9b94cc1a46006ea5bfbfee0e51f0fe" Nov 01 16:00:23.401407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.401386 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9"] Nov 01 16:00:23.448250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.448219 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r"] Nov 01 16:00:23.451441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.451419 4573 scope.go:122] "RemoveContainer" containerID="0788d5ccb3432602d3f3b9d9874a2a92a11164739bc85769517c1e707997326f" Nov 01 16:00:23.454393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.454369 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892480-7pr7r"] Nov 01 16:00:23.526523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.526501 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:23.572566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.572528 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-xks-ca\") pod \"287fbc1d-571f-41ae-afb9-251f2da0d568\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " Nov 01 16:00:23.572723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.572664 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-openshift-ca\") pod \"287fbc1d-571f-41ae-afb9-251f2da0d568\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " Nov 01 16:00:23.572815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.572797 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/287fbc1d-571f-41ae-afb9-251f2da0d568-kube-api-access-lj554\" (UniqueName: \"kubernetes.io/projected/287fbc1d-571f-41ae-afb9-251f2da0d568-kube-api-access-lj554\") pod \"287fbc1d-571f-41ae-afb9-251f2da0d568\" (UID: \"287fbc1d-571f-41ae-afb9-251f2da0d568\") " Nov 01 16:00:23.573119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.573092 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-openshift-ca" pod "287fbc1d-571f-41ae-afb9-251f2da0d568" (UID: "287fbc1d-571f-41ae-afb9-251f2da0d568"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:23.573581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.573554 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.574777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.574720 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-xks-ca" pod "287fbc1d-571f-41ae-afb9-251f2da0d568" (UID: "287fbc1d-571f-41ae-afb9-251f2da0d568"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:23.575302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.575278 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/287fbc1d-571f-41ae-afb9-251f2da0d568-kube-api-access-lj554" pod "287fbc1d-571f-41ae-afb9-251f2da0d568" (UID: "287fbc1d-571f-41ae-afb9-251f2da0d568"). InnerVolumeSpecName "kube-api-access-lj554". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:23.675356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.675307 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/287fbc1d-571f-41ae-afb9-251f2da0d568-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.675356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.675331 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj554\" (UniqueName: \"kubernetes.io/projected/287fbc1d-571f-41ae-afb9-251f2da0d568-kube-api-access-lj554\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:23.972927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:23.972846 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:00:24.047938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.047877 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:24.365931 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:24.365883 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:00:24.371085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.371062 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" Nov 01 16:00:24.371085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.371073 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f" event={"ID":"287fbc1d-571f-41ae-afb9-251f2da0d568","Type":"ContainerDied","Data":"29a444b1a9063c321911d446833bd529ca2e9e7df0c4ed74bf12461ac188dc85"} Nov 01 16:00:24.371297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.371128 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a444b1a9063c321911d446833bd529ca2e9e7df0c4ed74bf12461ac188dc85" Nov 01 16:00:24.378673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.378650 4573 generic.go:356] "Generic (PLEG): container finished" podID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerID="931dbdcd071197b8ca0f6bb468ebe37edb3bb08832968dba6897db53cad125e5" exitCode=60 Nov 01 16:00:24.378806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.378723 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" event={"ID":"a5e93ce3-bf81-4f68-b659-b96d0b1c0427","Type":"ContainerDied","Data":"931dbdcd071197b8ca0f6bb468ebe37edb3bb08832968dba6897db53cad125e5"} Nov 01 16:00:24.378865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.378824 4573 scope.go:122] "RemoveContainer" containerID="7c50603d789456ae7ec782ed58f9acc900587e3217bb8405fcd0977a79becd8a" Nov 01 16:00:24.379193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.379173 4573 scope.go:122] "RemoveContainer" containerID="931dbdcd071197b8ca0f6bb468ebe37edb3bb08832968dba6897db53cad125e5" Nov 01 16:00:24.379520 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:24.379495 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw_odh-ai-gateway-infra(a5e93ce3-bf81-4f68-b659-b96d0b1c0427)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" Nov 01 16:00:24.381378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.381353 4573 generic.go:356] "Generic (PLEG): container finished" podID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerID="9a5fb00953dad4b7359d5964322284f30bf807451b470788054ae59941909c78" exitCode=60 Nov 01 16:00:24.381488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.381431 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" event={"ID":"c08b4081-5132-4dc8-95b8-df8769d19e57","Type":"ContainerDied","Data":"9a5fb00953dad4b7359d5964322284f30bf807451b470788054ae59941909c78"} Nov 01 16:00:24.381855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.381833 4573 scope.go:122] "RemoveContainer" containerID="9a5fb00953dad4b7359d5964322284f30bf807451b470788054ae59941909c78" Nov 01 16:00:24.382387 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:24.382146 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892480-rqwd5_odh-ai-gateway-infra(c08b4081-5132-4dc8-95b8-df8769d19e57)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" Nov 01 16:00:24.384025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.383932 4573 generic.go:356] "Generic (PLEG): container finished" podID="6ab8c178-c15d-4009-a49d-611014f91f05" containerID="80d0c88fa969af725a83e9bf810dbd7a2367b4b8bcf9bf9ed0d1225a054d9a21" exitCode=0 Nov 01 16:00:24.384025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.383979 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" event={"ID":"6ab8c178-c15d-4009-a49d-611014f91f05","Type":"ContainerDied","Data":"80d0c88fa969af725a83e9bf810dbd7a2367b4b8bcf9bf9ed0d1225a054d9a21"} Nov 01 16:00:24.395319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.391621 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw"] Nov 01 16:00:24.396936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.396914 4573 scope.go:122] "RemoveContainer" containerID="0788d5ccb3432602d3f3b9d9874a2a92a11164739bc85769517c1e707997326f" Nov 01 16:00:24.469636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.469600 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3469177a-3810-4974-b737-66199dff79fd" path="/var/lib/kubelet/pods/3469177a-3810-4974-b737-66199dff79fd/volumes" Nov 01 16:00:24.524800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.524739 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:24.587516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.586633 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1605fbbc-8000-4cab-a969-b1a6eef41b6d-kube-api-access-gchkq\" (UniqueName: \"kubernetes.io/projected/1605fbbc-8000-4cab-a969-b1a6eef41b6d-kube-api-access-gchkq\") pod \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\" (UID: \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\") " Nov 01 16:00:24.587516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.586686 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1605fbbc-8000-4cab-a969-b1a6eef41b6d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1605fbbc-8000-4cab-a969-b1a6eef41b6d-maas-api-service-ca\") pod \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\" (UID: \"1605fbbc-8000-4cab-a969-b1a6eef41b6d\") " Nov 01 16:00:24.587913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.587868 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1605fbbc-8000-4cab-a969-b1a6eef41b6d-maas-api-service-ca" pod "1605fbbc-8000-4cab-a969-b1a6eef41b6d" (UID: "1605fbbc-8000-4cab-a969-b1a6eef41b6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:24.589641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.589615 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1605fbbc-8000-4cab-a969-b1a6eef41b6d-kube-api-access-gchkq" pod "1605fbbc-8000-4cab-a969-b1a6eef41b6d" (UID: "1605fbbc-8000-4cab-a969-b1a6eef41b6d"). InnerVolumeSpecName "kube-api-access-gchkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:24.618301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.618238 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:24.688337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.688297 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de4b893c-251a-445b-b9e7-a1c68571497d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4b893c-251a-445b-b9e7-a1c68571497d-maas-api-service-ca\") pod \"de4b893c-251a-445b-b9e7-a1c68571497d\" (UID: \"de4b893c-251a-445b-b9e7-a1c68571497d\") " Nov 01 16:00:24.688512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.688350 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de4b893c-251a-445b-b9e7-a1c68571497d-kube-api-access-hm8j5\" (UniqueName: \"kubernetes.io/projected/de4b893c-251a-445b-b9e7-a1c68571497d-kube-api-access-hm8j5\") pod \"de4b893c-251a-445b-b9e7-a1c68571497d\" (UID: \"de4b893c-251a-445b-b9e7-a1c68571497d\") " Nov 01 16:00:24.688706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.688680 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de4b893c-251a-445b-b9e7-a1c68571497d-maas-api-service-ca" pod "de4b893c-251a-445b-b9e7-a1c68571497d" (UID: "de4b893c-251a-445b-b9e7-a1c68571497d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:24.688786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.688705 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchkq\" (UniqueName: \"kubernetes.io/projected/1605fbbc-8000-4cab-a969-b1a6eef41b6d-kube-api-access-gchkq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:24.688786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.688724 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1605fbbc-8000-4cab-a969-b1a6eef41b6d-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:24.690556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.690533 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de4b893c-251a-445b-b9e7-a1c68571497d-kube-api-access-hm8j5" pod "de4b893c-251a-445b-b9e7-a1c68571497d" (UID: "de4b893c-251a-445b-b9e7-a1c68571497d"). InnerVolumeSpecName "kube-api-access-hm8j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:24.731680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.731652 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:24.790006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.789977 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de4b893c-251a-445b-b9e7-a1c68571497d-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:24.790006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:24.790001 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm8j5\" (UniqueName: \"kubernetes.io/projected/de4b893c-251a-445b-b9e7-a1c68571497d-kube-api-access-hm8j5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:25.427132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:25.427094 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" event={"ID":"1605fbbc-8000-4cab-a969-b1a6eef41b6d","Type":"ContainerDied","Data":"3a009fe932d3cf6d08ad122bbd565a12f16d0e7be0f45b01453156edcd27a70a"} Nov 01 16:00:25.427803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:25.427780 4573 scope.go:122] "RemoveContainer" containerID="7728e4083f4a8ea79bed40fcb63cb3089d5e6c92f3f6647f3cd3ab1f9cf9c064" Nov 01 16:00:25.428102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:25.428084 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9" Nov 01 16:00:25.436255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:25.436233 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5"] Nov 01 16:00:25.497820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:25.495049 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" Nov 01 16:00:25.499450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:25.498902 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6" event={"ID":"de4b893c-251a-445b-b9e7-a1c68571497d","Type":"ContainerDied","Data":"9a1d7b4407b6fd065836572b582aa32b3104923cb90963c726c7b41ccd6974bc"} Nov 01 16:00:25.576787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:25.571158 4573 scope.go:122] "RemoveContainer" containerID="61bf9540bd97e1758a78fa1c89e3b901f27c590a4341ce610ae0efc412e1fbae" Nov 01 16:00:26.022244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.001796 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6"] Nov 01 16:00:26.117976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.117942 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-4a083d-29892480-kbvp6"] Nov 01 16:00:26.134772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.130462 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9"] Nov 01 16:00:26.137223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.135862 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892480-td5j9"] Nov 01 16:00:26.203847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.203817 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:26.312763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.312712 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c08b4081-5132-4dc8-95b8-df8769d19e57-kube-api-access-875s2\" (UniqueName: \"kubernetes.io/projected/c08b4081-5132-4dc8-95b8-df8769d19e57-kube-api-access-875s2\") pod \"c08b4081-5132-4dc8-95b8-df8769d19e57\" (UID: \"c08b4081-5132-4dc8-95b8-df8769d19e57\") " Nov 01 16:00:26.317965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.317935 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c08b4081-5132-4dc8-95b8-df8769d19e57-kube-api-access-875s2" pod "c08b4081-5132-4dc8-95b8-df8769d19e57" (UID: "c08b4081-5132-4dc8-95b8-df8769d19e57"). InnerVolumeSpecName "kube-api-access-875s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:26.341969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.341948 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:26.346272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.346247 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:26.414800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.414388 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c08b4081-5132-4dc8-95b8-df8769d19e57-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08b4081-5132-4dc8-95b8-df8769d19e57-maas-api-service-ca\") pod \"c08b4081-5132-4dc8-95b8-df8769d19e57\" (UID: \"c08b4081-5132-4dc8-95b8-df8769d19e57\") " Nov 01 16:00:26.414800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.414504 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab8c178-c15d-4009-a49d-611014f91f05-kube-api-access-lqvxb\" (UniqueName: \"kubernetes.io/projected/6ab8c178-c15d-4009-a49d-611014f91f05-kube-api-access-lqvxb\") pod \"6ab8c178-c15d-4009-a49d-611014f91f05\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " Nov 01 16:00:26.414800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.414625 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-openshift-ca\") pod \"6ab8c178-c15d-4009-a49d-611014f91f05\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " Nov 01 16:00:26.414800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.414734 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-maas-api-service-ca\") pod \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\" (UID: \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\") " Nov 01 16:00:26.415182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.414830 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-kube-api-access-rz4rk\" (UniqueName: \"kubernetes.io/projected/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-kube-api-access-rz4rk\") pod \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\" (UID: \"a5e93ce3-bf81-4f68-b659-b96d0b1c0427\") " Nov 01 16:00:26.415182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.414879 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-xks-ca\") pod \"6ab8c178-c15d-4009-a49d-611014f91f05\" (UID: \"6ab8c178-c15d-4009-a49d-611014f91f05\") " Nov 01 16:00:26.425505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.415334 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-875s2\" (UniqueName: \"kubernetes.io/projected/c08b4081-5132-4dc8-95b8-df8769d19e57-kube-api-access-875s2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.425505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.417537 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-openshift-ca" pod "6ab8c178-c15d-4009-a49d-611014f91f05" (UID: "6ab8c178-c15d-4009-a49d-611014f91f05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:26.425505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.417701 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-maas-api-service-ca" pod "a5e93ce3-bf81-4f68-b659-b96d0b1c0427" (UID: "a5e93ce3-bf81-4f68-b659-b96d0b1c0427"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:26.425505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.418071 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c08b4081-5132-4dc8-95b8-df8769d19e57-maas-api-service-ca" pod "c08b4081-5132-4dc8-95b8-df8769d19e57" (UID: "c08b4081-5132-4dc8-95b8-df8769d19e57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:26.425505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.420780 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab8c178-c15d-4009-a49d-611014f91f05-kube-api-access-lqvxb" pod "6ab8c178-c15d-4009-a49d-611014f91f05" (UID: "6ab8c178-c15d-4009-a49d-611014f91f05"). InnerVolumeSpecName "kube-api-access-lqvxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:26.425505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.422591 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-kube-api-access-rz4rk" pod "a5e93ce3-bf81-4f68-b659-b96d0b1c0427" (UID: "a5e93ce3-bf81-4f68-b659-b96d0b1c0427"). InnerVolumeSpecName "kube-api-access-rz4rk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:26.425505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.423695 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-xks-ca" pod "6ab8c178-c15d-4009-a49d-611014f91f05" (UID: "6ab8c178-c15d-4009-a49d-611014f91f05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:26.517774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.517670 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.517774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.517712 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.517774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.517729 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz4rk\" (UniqueName: \"kubernetes.io/projected/a5e93ce3-bf81-4f68-b659-b96d0b1c0427-kube-api-access-rz4rk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.517774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.517766 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab8c178-c15d-4009-a49d-611014f91f05-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.517774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.517783 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c08b4081-5132-4dc8-95b8-df8769d19e57-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.518509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.517797 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqvxb\" (UniqueName: \"kubernetes.io/projected/6ab8c178-c15d-4009-a49d-611014f91f05-kube-api-access-lqvxb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:26.543085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.543005 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" path="/var/lib/kubelet/pods/1605fbbc-8000-4cab-a969-b1a6eef41b6d/volumes" Nov 01 16:00:26.543524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.543503 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" path="/var/lib/kubelet/pods/de4b893c-251a-445b-b9e7-a1c68571497d/volumes" Nov 01 16:00:26.553327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.547197 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" event={"ID":"c08b4081-5132-4dc8-95b8-df8769d19e57","Type":"ContainerDied","Data":"1c416bfa42fe6d57dc360ae6e45de22a26469060f7717a9ffbef9487e88f6812"} Nov 01 16:00:26.553327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.547256 4573 scope.go:122] "RemoveContainer" containerID="9a5fb00953dad4b7359d5964322284f30bf807451b470788054ae59941909c78" Nov 01 16:00:26.553327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.547357 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5" Nov 01 16:00:26.553685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.553370 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" event={"ID":"6ab8c178-c15d-4009-a49d-611014f91f05","Type":"ContainerDied","Data":"3803d199676b3e692ef5ff46cc1c2064c706838fe8283db2ffced33cc00127f2"} Nov 01 16:00:26.553685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.553411 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3803d199676b3e692ef5ff46cc1c2064c706838fe8283db2ffced33cc00127f2" Nov 01 16:00:26.553685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.553468 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d" Nov 01 16:00:26.560959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.560923 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" event={"ID":"a5e93ce3-bf81-4f68-b659-b96d0b1c0427","Type":"ContainerDied","Data":"948f774383fe44495a1970da31590e1ae335fb0c300806e884a944e6684712db"} Nov 01 16:00:26.560959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.560950 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw" Nov 01 16:00:26.629804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.625790 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5"] Nov 01 16:00:26.647208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.637066 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892480-rqwd5"] Nov 01 16:00:26.651578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.651553 4573 scope.go:122] "RemoveContainer" containerID="931dbdcd071197b8ca0f6bb468ebe37edb3bb08832968dba6897db53cad125e5" Nov 01 16:00:26.652209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.652162 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw"] Nov 01 16:00:26.652314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.652208 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:26.654949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:26.654926 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892480-ghhlw"] Nov 01 16:00:27.631256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:27.631220 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:28.458267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.458235 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" path="/var/lib/kubelet/pods/a5e93ce3-bf81-4f68-b659-b96d0b1c0427/volumes" Nov 01 16:00:28.458649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.458636 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" path="/var/lib/kubelet/pods/c08b4081-5132-4dc8-95b8-df8769d19e57/volumes" Nov 01 16:00:28.465726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.465690 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv"] Nov 01 16:00:28.466204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466190 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.466263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466205 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.466263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466223 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.466263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466228 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.466263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466237 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.466263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466242 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.466263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466250 4573 cpu_manager.go:414] "containerMap: removing container" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.466263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466254 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466268 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466273 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466287 4573 cpu_manager.go:414] "containerMap: removing container" podUID="287fbc1d-571f-41ae-afb9-251f2da0d568" containerName="revoke-keys" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466291 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="287fbc1d-571f-41ae-afb9-251f2da0d568" containerName="revoke-keys" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466298 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466302 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466309 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466313 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466322 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466326 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466331 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466336 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466345 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466349 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466358 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6ab8c178-c15d-4009-a49d-611014f91f05" containerName="revoke-keys" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466362 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab8c178-c15d-4009-a49d-611014f91f05" containerName="revoke-keys" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466369 4573 cpu_manager.go:414] "containerMap: removing container" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466374 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466443 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab8c178-c15d-4009-a49d-611014f91f05" containerName="revoke-keys" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466452 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466460 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466466 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466472 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466478 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.466474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466482 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.467208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466488 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.467208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466495 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="287fbc1d-571f-41ae-afb9-251f2da0d568" containerName="revoke-keys" Nov 01 16:00:28.467208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466501 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.467208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466506 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.467208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466511 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.467208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466516 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.467208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466523 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1605fbbc-8000-4cab-a969-b1a6eef41b6d" containerName="cleanup" Nov 01 16:00:28.467208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.466529 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.470042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.470023 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.477287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.477267 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-164db6-openshift-default-dockercfg-mt9cn\"" Nov 01 16:00:28.482265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.482243 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv"] Nov 01 16:00:28.554637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.554599 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.554817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.554658 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.554817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.554732 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.554925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.554860 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.554925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.554913 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.555017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.554946 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.555106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.555064 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.555214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.555136 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkhr4\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-kube-api-access-nkhr4\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.555214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.555163 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.582171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.582147 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:28.656216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656176 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.656731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656256 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.656731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656286 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.656731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656386 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.656731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656423 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.656731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656467 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.656731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656568 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.656731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656603 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkhr4\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-kube-api-access-nkhr4\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.656731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656636 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.657204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.656983 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.657204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.657040 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.657307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.657294 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.657730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.657702 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.657949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.657725 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.659351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.659329 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.660045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.660023 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.667176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.667146 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.670673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.670638 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkhr4\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-kube-api-access-nkhr4\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.784780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.784684 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:28.915424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.915381 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k"] Nov 01 16:00:28.915959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.915940 4573 cpu_manager.go:414] "containerMap: removing container" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.915959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.915956 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.916133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.915986 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.916133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.915992 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.916133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.915997 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.916133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.916003 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5e93ce3-bf81-4f68-b659-b96d0b1c0427" containerName="cleanup" Nov 01 16:00:28.916133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.916025 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.916133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.916030 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" containerName="cleanup" Nov 01 16:00:28.916133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.916106 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="de4b893c-251a-445b-b9e7-a1c68571497d" containerName="cleanup" Nov 01 16:00:28.916133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.916119 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3469177a-3810-4974-b737-66199dff79fd" containerName="cleanup" Nov 01 16:00:28.920639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.920617 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.930955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.930932 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k"] Nov 01 16:00:28.959236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959202 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.959375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959318 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.959375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959341 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.959487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959405 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p475h\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-kube-api-access-p475h\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.959487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959442 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4247c96-f138-451a-9a73-305270e6a4a0-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.959487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959480 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4247c96-f138-451a-9a73-305270e6a4a0-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.959642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959537 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.959642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959587 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:28.959739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:28.959670 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060393 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060443 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060528 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060554 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060607 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p475h\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-kube-api-access-p475h\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060649 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4247c96-f138-451a-9a73-305270e6a4a0-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060708 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4247c96-f138-451a-9a73-305270e6a4a0-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060740 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060844 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060853 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.060979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.060892 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.061361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.061154 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.061361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.061155 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.061471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.061450 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4247c96-f138-451a-9a73-305270e6a4a0-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.062989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.062965 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.063154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.063138 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4247c96-f138-451a-9a73-305270e6a4a0-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.069454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.069435 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.069581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.069560 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv"] Nov 01 16:00:29.070056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.070037 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p475h\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-kube-api-access-p475h\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.070429 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:29.070406 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dcbf4a1_4324_446c_911a_b0836fb21ba5.slice/crio-c3e1db775020390b825f4c5378fdffe99af6e6ef507f921a757da5427a0eda0b WatchSource:0}: Error finding container c3e1db775020390b825f4c5378fdffe99af6e6ef507f921a757da5427a0eda0b: Status 404 returned error can't find the container with id c3e1db775020390b825f4c5378fdffe99af6e6ef507f921a757da5427a0eda0b Nov 01 16:00:29.072800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.072773 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.072855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.072831 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.072888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.072862 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.239042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.239008 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:29.461398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.461366 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:29.520580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.520554 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k"] Nov 01 16:00:29.521417 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:29.521393 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4247c96_f138_451a_9a73_305270e6a4a0.slice/crio-d9abd883124758fd558cddc8d0710946b28c8bfdbd1f24b08a0d5d4887ae9b91 WatchSource:0}: Error finding container d9abd883124758fd558cddc8d0710946b28c8bfdbd1f24b08a0d5d4887ae9b91: Status 404 returned error can't find the container with id d9abd883124758fd558cddc8d0710946b28c8bfdbd1f24b08a0d5d4887ae9b91 Nov 01 16:00:29.525780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.524139 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.525780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.524217 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.525780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.524262 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:29.587436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.587402 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" event={"ID":"5dcbf4a1-4324-446c-911a-b0836fb21ba5","Type":"ContainerStarted","Data":"aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049"} Nov 01 16:00:29.587574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.587469 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" event={"ID":"5dcbf4a1-4324-446c-911a-b0836fb21ba5","Type":"ContainerStarted","Data":"c3e1db775020390b825f4c5378fdffe99af6e6ef507f921a757da5427a0eda0b"} Nov 01 16:00:29.588854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.588822 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" event={"ID":"e4247c96-f138-451a-9a73-305270e6a4a0","Type":"ContainerStarted","Data":"d9abd883124758fd558cddc8d0710946b28c8bfdbd1f24b08a0d5d4887ae9b91"} Nov 01 16:00:29.607996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.607945 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podStartSLOduration=1.607927787 podStartE2EDuration="1.607927787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:29.604394309 +0000 UTC m=+767.307560247" watchObservedRunningTime="2026-11-01 16:00:29.607927787 +0000 UTC m=+767.311093726" Nov 01 16:00:29.785294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:29.785199 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:30.209775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:30.209333 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:30.434105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:30.434065 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:30.619881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:30.619847 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" event={"ID":"e4247c96-f138-451a-9a73-305270e6a4a0","Type":"ContainerStarted","Data":"30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a"} Nov 01 16:00:30.642497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:30.642441 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podStartSLOduration=2.642427795 podStartE2EDuration="2.642427795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:30.641292027 +0000 UTC m=+768.344457964" watchObservedRunningTime="2026-11-01 16:00:30.642427795 +0000 UTC m=+768.345593731" Nov 01 16:00:30.794776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:30.792855 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:00:30.794776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:30.792919 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 16:00:31.240213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.240174 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:31.241315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.241288 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:00:31.241388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.241348 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 16:00:31.494837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.490629 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:31.516077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.516038 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h"] Nov 01 16:00:31.521636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.521612 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.530766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.530728 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h"] Nov 01 16:00:31.532860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.532839 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:31.532976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.532924 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:00:31.537780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.537738 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs"] Nov 01 16:00:31.545714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.545689 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs"] Nov 01 16:00:31.545837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.545817 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.735772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.735724 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.735944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.735814 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktg7\" (UniqueName: \"kubernetes.io/projected/67df2dc9-d8e1-4566-a322-7895480f22ae-kube-api-access-kktg7\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.736025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.735937 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.736025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.735962 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.736025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.735981 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.736172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.736069 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9wxm\" (UniqueName: \"kubernetes.io/projected/9e39bf32-1293-4995-90a6-53b547f4aca2-kube-api-access-v9wxm\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.801840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.800774 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:00:31.801840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.800850 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 16:00:31.837788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.837715 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.837963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.837822 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktg7\" (UniqueName: \"kubernetes.io/projected/67df2dc9-d8e1-4566-a322-7895480f22ae-kube-api-access-kktg7\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.838074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.838048 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.838125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.838091 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.838157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.838125 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.838237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.838221 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9wxm\" (UniqueName: \"kubernetes.io/projected/9e39bf32-1293-4995-90a6-53b547f4aca2-kube-api-access-v9wxm\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.838661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.838637 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.838866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.838843 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.841352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.841115 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.843219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.843198 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.845649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.845625 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktg7\" (UniqueName: \"kubernetes.io/projected/67df2dc9-d8e1-4566-a322-7895480f22ae-kube-api-access-kktg7\") pod \"maas-api-revoke-sub-045ca2d2-trv4h\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.845806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.845774 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:31.846269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.846251 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9wxm\" (UniqueName: \"kubernetes.io/projected/9e39bf32-1293-4995-90a6-53b547f4aca2-kube-api-access-v9wxm\") pod \"maas-api-revoke-sub-e1897a7f-6vmzs\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:31.895132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:31.895099 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:32.239496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.239468 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:00:32.239587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.239514 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 16:00:32.241665 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:32.241641 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67df2dc9_d8e1_4566_a322_7895480f22ae.slice/crio-fdf97ed27905be231762690ea03a7c0e314b111ac13c3d3a6ed3790f28e51536 WatchSource:0}: Error finding container fdf97ed27905be231762690ea03a7c0e314b111ac13c3d3a6ed3790f28e51536: Status 404 returned error can't find the container with id fdf97ed27905be231762690ea03a7c0e314b111ac13c3d3a6ed3790f28e51536 Nov 01 16:00:32.243068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.243029 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h"] Nov 01 16:00:32.253577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.253557 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs"] Nov 01 16:00:32.255025 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:32.255003 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e39bf32_1293_4995_90a6_53b547f4aca2.slice/crio-52b02d424bf5c22cccdf8f5ec4158e95c8e8117871f7909fae00752c9fd6d51a WatchSource:0}: Error finding container 52b02d424bf5c22cccdf8f5ec4158e95c8e8117871f7909fae00752c9fd6d51a: Status 404 returned error can't find the container with id 52b02d424bf5c22cccdf8f5ec4158e95c8e8117871f7909fae00752c9fd6d51a Nov 01 16:00:32.651939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.651901 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" event={"ID":"9e39bf32-1293-4995-90a6-53b547f4aca2","Type":"ContainerStarted","Data":"52b02d424bf5c22cccdf8f5ec4158e95c8e8117871f7909fae00752c9fd6d51a"} Nov 01 16:00:32.653195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.653169 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" event={"ID":"67df2dc9-d8e1-4566-a322-7895480f22ae","Type":"ContainerStarted","Data":"fdf97ed27905be231762690ea03a7c0e314b111ac13c3d3a6ed3790f28e51536"} Nov 01 16:00:32.674981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.674956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:32.751364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.751334 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n"] Nov 01 16:00:32.756074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.756052 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.779793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.761918 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n"] Nov 01 16:00:32.823774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.820057 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:00:32.823774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.820117 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 16:00:32.851865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.851835 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.852112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.852085 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxh8s\" (UniqueName: \"kubernetes.io/projected/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-kube-api-access-kxh8s\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.852223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.852208 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.953336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.953259 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxh8s\" (UniqueName: \"kubernetes.io/projected/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-kube-api-access-kxh8s\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.953336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.953333 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.953513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.953367 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.954043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.954015 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.956154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.956130 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:32.961437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:32.961419 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxh8s\" (UniqueName: \"kubernetes.io/projected/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-kube-api-access-kxh8s\") pod \"maas-api-revoke-sub-aa5f90da-p955n\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:33.046851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.046812 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7"] Nov 01 16:00:33.058214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.056969 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.058214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.058103 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7"] Nov 01 16:00:33.133226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.133194 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:33.157728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.157705 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.157862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.157738 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfdn\" (UniqueName: \"kubernetes.io/projected/c4a544fe-3957-48ef-a092-1cc47533cf92-kube-api-access-8zfdn\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.157923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.157874 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.240088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.240051 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:00:33.240208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.240117 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 16:00:33.259373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.259342 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.259488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.259392 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfdn\" (UniqueName: \"kubernetes.io/projected/c4a544fe-3957-48ef-a092-1cc47533cf92-kube-api-access-8zfdn\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.259488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.259476 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.260431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.260409 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.267802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.261843 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.318804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.313557 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfdn\" (UniqueName: \"kubernetes.io/projected/c4a544fe-3957-48ef-a092-1cc47533cf92-kube-api-access-8zfdn\") pod \"maas-api-revoke-sub-981ffd63-w6jc7\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.428960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.428914 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:33.460471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.460443 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n"] Nov 01 16:00:33.460810 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:33.460787 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bc0b55_74f0_4bf6_a338_24565fe9fa2c.slice/crio-f1b0520f16817f617b119d19afd6dd1e927fad4ed49f1bafed70ca25b55592a1 WatchSource:0}: Error finding container f1b0520f16817f617b119d19afd6dd1e927fad4ed49f1bafed70ca25b55592a1: Status 404 returned error can't find the container with id f1b0520f16817f617b119d19afd6dd1e927fad4ed49f1bafed70ca25b55592a1 Nov 01 16:00:33.661626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.661588 4573 generic.go:356] "Generic (PLEG): container finished" podID="9e39bf32-1293-4995-90a6-53b547f4aca2" containerID="2bf8385e90728f7f244f5d989b217718f0258f592c3c2d0b096da7687f202a4c" exitCode=0 Nov 01 16:00:33.669872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.661674 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" event={"ID":"9e39bf32-1293-4995-90a6-53b547f4aca2","Type":"ContainerDied","Data":"2bf8385e90728f7f244f5d989b217718f0258f592c3c2d0b096da7687f202a4c"} Nov 01 16:00:33.719127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.719083 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" event={"ID":"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c","Type":"ContainerStarted","Data":"f1b0520f16817f617b119d19afd6dd1e927fad4ed49f1bafed70ca25b55592a1"} Nov 01 16:00:33.719329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.719155 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:33.721834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.721565 4573 generic.go:356] "Generic (PLEG): container finished" podID="67df2dc9-d8e1-4566-a322-7895480f22ae" containerID="443af5c3b62fef7e13989b9ae0066d27db684af7d528966168894c28135098ef" exitCode=0 Nov 01 16:00:33.721834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.721705 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" event={"ID":"67df2dc9-d8e1-4566-a322-7895480f22ae","Type":"ContainerDied","Data":"443af5c3b62fef7e13989b9ae0066d27db684af7d528966168894c28135098ef"} Nov 01 16:00:33.807837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.799001 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:00:33.807837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.799074 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 16:00:33.822773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:33.820141 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7"] Nov 01 16:00:34.024208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.024110 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:34.239928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.239656 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:00:34.239928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.239715 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 16:00:34.731141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.731100 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" event={"ID":"c4a544fe-3957-48ef-a092-1cc47533cf92","Type":"ContainerStarted","Data":"5745bacde7ccdadd2266b0e9224e662f6fc2aa1f1ed93b0589bfbed0803879d9"} Nov 01 16:00:34.731332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.731246 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" event={"ID":"c4a544fe-3957-48ef-a092-1cc47533cf92","Type":"ContainerStarted","Data":"46712de0a3ada295497cf3433491f14990b475cb618417b6b2054efa2b365761"} Nov 01 16:00:34.733146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.733120 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" event={"ID":"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c","Type":"ContainerStarted","Data":"d9d2cf8d9859f9a08d0cba9d692ccbd89979b00ef008da18cac1b7c4a6c518ec"} Nov 01 16:00:34.749652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.749541 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" podStartSLOduration=1.749527843 podStartE2EDuration="1.749527843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:34.746294084 +0000 UTC m=+772.449460035" watchObservedRunningTime="2026-11-01 16:00:34.749527843 +0000 UTC m=+772.452693796" Nov 01 16:00:34.776442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.771130 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" podStartSLOduration=2.771112918 podStartE2EDuration="2.771112918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:34.768108305 +0000 UTC m=+772.471274242" watchObservedRunningTime="2026-11-01 16:00:34.771112918 +0000 UTC m=+772.474278856" Nov 01 16:00:34.816774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.796363 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:00:34.816774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:34.796421 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 16:00:35.076775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.070932 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:35.123323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.123281 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:35.141702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.140954 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e39bf32-1293-4995-90a6-53b547f4aca2-kube-api-access-v9wxm\" (UniqueName: \"kubernetes.io/projected/9e39bf32-1293-4995-90a6-53b547f4aca2-kube-api-access-v9wxm\") pod \"9e39bf32-1293-4995-90a6-53b547f4aca2\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " Nov 01 16:00:35.141702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.141023 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-openshift-ca\") pod \"9e39bf32-1293-4995-90a6-53b547f4aca2\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " Nov 01 16:00:35.141702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.141080 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-xks-ca\") pod \"9e39bf32-1293-4995-90a6-53b547f4aca2\" (UID: \"9e39bf32-1293-4995-90a6-53b547f4aca2\") " Nov 01 16:00:35.142128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.142102 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-openshift-ca" pod "9e39bf32-1293-4995-90a6-53b547f4aca2" (UID: "9e39bf32-1293-4995-90a6-53b547f4aca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:35.143624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.143581 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e39bf32-1293-4995-90a6-53b547f4aca2-kube-api-access-v9wxm" pod "9e39bf32-1293-4995-90a6-53b547f4aca2" (UID: "9e39bf32-1293-4995-90a6-53b547f4aca2"). InnerVolumeSpecName "kube-api-access-v9wxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:35.143714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.143682 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-xks-ca" pod "9e39bf32-1293-4995-90a6-53b547f4aca2" (UID: "9e39bf32-1293-4995-90a6-53b547f4aca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:35.240413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.239925 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:35.241369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.241327 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 01 16:00:35.241502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.241384 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 01 16:00:35.242450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.242418 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9wxm\" (UniqueName: \"kubernetes.io/projected/9e39bf32-1293-4995-90a6-53b547f4aca2-kube-api-access-v9wxm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:35.242569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.242454 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:35.242569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.242499 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e39bf32-1293-4995-90a6-53b547f4aca2-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:35.343784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.343723 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-xks-ca\") pod \"67df2dc9-d8e1-4566-a322-7895480f22ae\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " Nov 01 16:00:35.343784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.343778 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67df2dc9-d8e1-4566-a322-7895480f22ae-kube-api-access-kktg7\" (UniqueName: \"kubernetes.io/projected/67df2dc9-d8e1-4566-a322-7895480f22ae-kube-api-access-kktg7\") pod \"67df2dc9-d8e1-4566-a322-7895480f22ae\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " Nov 01 16:00:35.344101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.343856 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-openshift-ca\") pod \"67df2dc9-d8e1-4566-a322-7895480f22ae\" (UID: \"67df2dc9-d8e1-4566-a322-7895480f22ae\") " Nov 01 16:00:35.344260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.344233 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-openshift-ca" pod "67df2dc9-d8e1-4566-a322-7895480f22ae" (UID: "67df2dc9-d8e1-4566-a322-7895480f22ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:35.344403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.344384 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:35.345913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.345883 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-xks-ca" pod "67df2dc9-d8e1-4566-a322-7895480f22ae" (UID: "67df2dc9-d8e1-4566-a322-7895480f22ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:35.345913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.345893 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67df2dc9-d8e1-4566-a322-7895480f22ae-kube-api-access-kktg7" pod "67df2dc9-d8e1-4566-a322-7895480f22ae" (UID: "67df2dc9-d8e1-4566-a322-7895480f22ae"). InnerVolumeSpecName "kube-api-access-kktg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:35.445332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.445295 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67df2dc9-d8e1-4566-a322-7895480f22ae-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:35.445332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.445326 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kktg7\" (UniqueName: \"kubernetes.io/projected/67df2dc9-d8e1-4566-a322-7895480f22ae-kube-api-access-kktg7\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:35.743677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.743646 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" Nov 01 16:00:35.743891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.743653 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h" event={"ID":"67df2dc9-d8e1-4566-a322-7895480f22ae","Type":"ContainerDied","Data":"fdf97ed27905be231762690ea03a7c0e314b111ac13c3d3a6ed3790f28e51536"} Nov 01 16:00:35.743891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.743839 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdf97ed27905be231762690ea03a7c0e314b111ac13c3d3a6ed3790f28e51536" Nov 01 16:00:35.745514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.745483 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" event={"ID":"9e39bf32-1293-4995-90a6-53b547f4aca2","Type":"ContainerDied","Data":"52b02d424bf5c22cccdf8f5ec4158e95c8e8117871f7909fae00752c9fd6d51a"} Nov 01 16:00:35.745514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.745506 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs" Nov 01 16:00:35.745724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.745534 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52b02d424bf5c22cccdf8f5ec4158e95c8e8117871f7909fae00752c9fd6d51a" Nov 01 16:00:35.747480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.747446 4573 generic.go:356] "Generic (PLEG): container finished" podID="c4a544fe-3957-48ef-a092-1cc47533cf92" containerID="5745bacde7ccdadd2266b0e9224e662f6fc2aa1f1ed93b0589bfbed0803879d9" exitCode=0 Nov 01 16:00:35.747603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.747562 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" event={"ID":"c4a544fe-3957-48ef-a092-1cc47533cf92","Type":"ContainerDied","Data":"5745bacde7ccdadd2266b0e9224e662f6fc2aa1f1ed93b0589bfbed0803879d9"} Nov 01 16:00:35.752520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.752452 4573 generic.go:356] "Generic (PLEG): container finished" podID="f5bc0b55-74f0-4bf6-a338-24565fe9fa2c" containerID="d9d2cf8d9859f9a08d0cba9d692ccbd89979b00ef008da18cac1b7c4a6c518ec" exitCode=0 Nov 01 16:00:35.752520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.752489 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" event={"ID":"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c","Type":"ContainerDied","Data":"d9d2cf8d9859f9a08d0cba9d692ccbd89979b00ef008da18cac1b7c4a6c518ec"} Nov 01 16:00:35.785001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.784973 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Nov 01 16:00:35.785142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:35.785025 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Nov 01 16:00:36.043165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.043074 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:36.079184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.079151 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w"] Nov 01 16:00:36.080065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.080042 4573 cpu_manager.go:414] "containerMap: removing container" podUID="67df2dc9-d8e1-4566-a322-7895480f22ae" containerName="revoke-keys" Nov 01 16:00:36.080163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.080066 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67df2dc9-d8e1-4566-a322-7895480f22ae" containerName="revoke-keys" Nov 01 16:00:36.080163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.080104 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9e39bf32-1293-4995-90a6-53b547f4aca2" containerName="revoke-keys" Nov 01 16:00:36.080163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.080112 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e39bf32-1293-4995-90a6-53b547f4aca2" containerName="revoke-keys" Nov 01 16:00:36.080337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.080268 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="67df2dc9-d8e1-4566-a322-7895480f22ae" containerName="revoke-keys" Nov 01 16:00:36.080337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.080299 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e39bf32-1293-4995-90a6-53b547f4aca2" containerName="revoke-keys" Nov 01 16:00:36.088520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.087856 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.091912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.090774 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w"] Nov 01 16:00:36.173179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.172623 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l98pq\" (UniqueName: \"kubernetes.io/projected/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-kube-api-access-l98pq\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.174969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.173892 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.175183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.175164 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.276486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.276448 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98pq\" (UniqueName: \"kubernetes.io/projected/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-kube-api-access-l98pq\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.276736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.276710 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.276976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.276947 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.279006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.278976 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.279882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.279862 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.286412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.286379 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l98pq\" (UniqueName: \"kubernetes.io/projected/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-kube-api-access-l98pq\") pod \"maas-api-revoke-sub-f41548c3-pwv7w\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.408001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.407965 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:36.410117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.408686 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:36.410117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.409790 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:00:36.411108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.411085 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:36.712412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.712380 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w"] Nov 01 16:00:36.714665 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:36.714636 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0268b3d0_bccd_477a_96bf_91ddd8f89bc1.slice/crio-0283f4257c5bc6404abfde7afe8a745997f74272021089f316d168b31e6cd24b WatchSource:0}: Error finding container 0283f4257c5bc6404abfde7afe8a745997f74272021089f316d168b31e6cd24b: Status 404 returned error can't find the container with id 0283f4257c5bc6404abfde7afe8a745997f74272021089f316d168b31e6cd24b Nov 01 16:00:36.762390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.762360 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" event={"ID":"0268b3d0-bccd-477a-96bf-91ddd8f89bc1","Type":"ContainerStarted","Data":"0283f4257c5bc6404abfde7afe8a745997f74272021089f316d168b31e6cd24b"} Nov 01 16:00:36.789995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.789971 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:36.790227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.790211 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:36.790789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.790771 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:00:36.973015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:36.972942 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:37.006997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.006970 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:37.038700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.038676 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:37.089507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.089482 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a544fe-3957-48ef-a092-1cc47533cf92-kube-api-access-8zfdn\" (UniqueName: \"kubernetes.io/projected/c4a544fe-3957-48ef-a092-1cc47533cf92-kube-api-access-8zfdn\") pod \"c4a544fe-3957-48ef-a092-1cc47533cf92\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " Nov 01 16:00:37.089865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.089598 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-openshift-ca\") pod \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " Nov 01 16:00:37.089865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.089639 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-openshift-ca\") pod \"c4a544fe-3957-48ef-a092-1cc47533cf92\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " Nov 01 16:00:37.089865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.089737 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-kube-api-access-kxh8s\" (UniqueName: \"kubernetes.io/projected/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-kube-api-access-kxh8s\") pod \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " Nov 01 16:00:37.089865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.089831 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-xks-ca\") pod \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\" (UID: \"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c\") " Nov 01 16:00:37.090065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.089878 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-xks-ca\") pod \"c4a544fe-3957-48ef-a092-1cc47533cf92\" (UID: \"c4a544fe-3957-48ef-a092-1cc47533cf92\") " Nov 01 16:00:37.090065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.089966 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-openshift-ca" pod "f5bc0b55-74f0-4bf6-a338-24565fe9fa2c" (UID: "f5bc0b55-74f0-4bf6-a338-24565fe9fa2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:37.090065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.089973 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-openshift-ca" pod "c4a544fe-3957-48ef-a092-1cc47533cf92" (UID: "c4a544fe-3957-48ef-a092-1cc47533cf92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:37.090396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.090375 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:37.090466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.090406 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:37.091472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.091451 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a544fe-3957-48ef-a092-1cc47533cf92-kube-api-access-8zfdn" pod "c4a544fe-3957-48ef-a092-1cc47533cf92" (UID: "c4a544fe-3957-48ef-a092-1cc47533cf92"). InnerVolumeSpecName "kube-api-access-8zfdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:37.092012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.091988 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-xks-ca" pod "c4a544fe-3957-48ef-a092-1cc47533cf92" (UID: "c4a544fe-3957-48ef-a092-1cc47533cf92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:37.092124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.092101 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-xks-ca" pod "f5bc0b55-74f0-4bf6-a338-24565fe9fa2c" (UID: "f5bc0b55-74f0-4bf6-a338-24565fe9fa2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:37.092168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.092109 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-kube-api-access-kxh8s" pod "f5bc0b55-74f0-4bf6-a338-24565fe9fa2c" (UID: "f5bc0b55-74f0-4bf6-a338-24565fe9fa2c"). InnerVolumeSpecName "kube-api-access-kxh8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:37.191524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.191486 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxh8s\" (UniqueName: \"kubernetes.io/projected/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-kube-api-access-kxh8s\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:37.191524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.191515 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:37.191524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.191528 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4a544fe-3957-48ef-a092-1cc47533cf92-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:37.191796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.191541 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zfdn\" (UniqueName: \"kubernetes.io/projected/c4a544fe-3957-48ef-a092-1cc47533cf92-kube-api-access-8zfdn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:37.682996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.682948 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:37.778561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.778328 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" Nov 01 16:00:37.778561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.778387 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7" event={"ID":"c4a544fe-3957-48ef-a092-1cc47533cf92","Type":"ContainerDied","Data":"46712de0a3ada295497cf3433491f14990b475cb618417b6b2054efa2b365761"} Nov 01 16:00:37.778561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.778460 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46712de0a3ada295497cf3433491f14990b475cb618417b6b2054efa2b365761" Nov 01 16:00:37.781461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.781443 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" Nov 01 16:00:37.781566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.781443 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n" event={"ID":"f5bc0b55-74f0-4bf6-a338-24565fe9fa2c","Type":"ContainerDied","Data":"f1b0520f16817f617b119d19afd6dd1e927fad4ed49f1bafed70ca25b55592a1"} Nov 01 16:00:37.781632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.781565 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1b0520f16817f617b119d19afd6dd1e927fad4ed49f1bafed70ca25b55592a1" Nov 01 16:00:37.783795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:37.783774 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" event={"ID":"0268b3d0-bccd-477a-96bf-91ddd8f89bc1","Type":"ContainerStarted","Data":"63c8e71cc0e3cfeb3b384f253b85487f9f97074b47dd5f5d1dd3c84832dfdd89"} Nov 01 16:00:38.478707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:38.478676 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:38.791881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:38.791787 4573 generic.go:356] "Generic (PLEG): container finished" podID="0268b3d0-bccd-477a-96bf-91ddd8f89bc1" containerID="63c8e71cc0e3cfeb3b384f253b85487f9f97074b47dd5f5d1dd3c84832dfdd89" exitCode=0 Nov 01 16:00:38.791881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:38.791854 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" event={"ID":"0268b3d0-bccd-477a-96bf-91ddd8f89bc1","Type":"ContainerDied","Data":"63c8e71cc0e3cfeb3b384f253b85487f9f97074b47dd5f5d1dd3c84832dfdd89"} Nov 01 16:00:38.942365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:38.942338 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:39.014714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.014685 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-kube-api-access-l98pq\" (UniqueName: \"kubernetes.io/projected/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-kube-api-access-l98pq\") pod \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " Nov 01 16:00:39.014890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.014769 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-xks-ca\") pod \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " Nov 01 16:00:39.014890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.014854 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-openshift-ca\") pod \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\" (UID: \"0268b3d0-bccd-477a-96bf-91ddd8f89bc1\") " Nov 01 16:00:39.015264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.015232 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-openshift-ca" pod "0268b3d0-bccd-477a-96bf-91ddd8f89bc1" (UID: "0268b3d0-bccd-477a-96bf-91ddd8f89bc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:39.015409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.015390 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:39.017366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.017343 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-kube-api-access-l98pq" pod "0268b3d0-bccd-477a-96bf-91ddd8f89bc1" (UID: "0268b3d0-bccd-477a-96bf-91ddd8f89bc1"). InnerVolumeSpecName "kube-api-access-l98pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:39.023087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.023064 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-xks-ca" pod "0268b3d0-bccd-477a-96bf-91ddd8f89bc1" (UID: "0268b3d0-bccd-477a-96bf-91ddd8f89bc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:39.116453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.116428 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l98pq\" (UniqueName: \"kubernetes.io/projected/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-kube-api-access-l98pq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:39.116453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.116456 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0268b3d0-bccd-477a-96bf-91ddd8f89bc1-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:39.321989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.321955 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:39.799546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.799519 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" Nov 01 16:00:39.800003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.799519 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w" event={"ID":"0268b3d0-bccd-477a-96bf-91ddd8f89bc1","Type":"ContainerDied","Data":"0283f4257c5bc6404abfde7afe8a745997f74272021089f316d168b31e6cd24b"} Nov 01 16:00:39.800003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:39.799636 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0283f4257c5bc6404abfde7afe8a745997f74272021089f316d168b31e6cd24b" Nov 01 16:00:40.049479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:40.049444 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:41.567773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:41.566801 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:42.688384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:42.688344 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:43.449733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:43.449688 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:44.362603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:44.362566 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:44.466456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:44.466425 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:45.095080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.095048 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:45.201505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.201475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:45.414804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.414774 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd"] Nov 01 16:00:45.415344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415328 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f5bc0b55-74f0-4bf6-a338-24565fe9fa2c" containerName="revoke-keys" Nov 01 16:00:45.415417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415345 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5bc0b55-74f0-4bf6-a338-24565fe9fa2c" containerName="revoke-keys" Nov 01 16:00:45.415417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415378 4573 cpu_manager.go:414] "containerMap: removing container" podUID="0268b3d0-bccd-477a-96bf-91ddd8f89bc1" containerName="revoke-keys" Nov 01 16:00:45.415417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415383 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0268b3d0-bccd-477a-96bf-91ddd8f89bc1" containerName="revoke-keys" Nov 01 16:00:45.415417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415394 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c4a544fe-3957-48ef-a092-1cc47533cf92" containerName="revoke-keys" Nov 01 16:00:45.415417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415400 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a544fe-3957-48ef-a092-1cc47533cf92" containerName="revoke-keys" Nov 01 16:00:45.415576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415493 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a544fe-3957-48ef-a092-1cc47533cf92" containerName="revoke-keys" Nov 01 16:00:45.415576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415503 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5bc0b55-74f0-4bf6-a338-24565fe9fa2c" containerName="revoke-keys" Nov 01 16:00:45.415576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.415516 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="0268b3d0-bccd-477a-96bf-91ddd8f89bc1" containerName="revoke-keys" Nov 01 16:00:45.420572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.420553 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.425618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.425594 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:00:45.426455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.426433 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:00:45.427778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.427604 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd"] Nov 01 16:00:45.579201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.579166 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.579409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.579368 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.579496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.579413 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g565g\" (UniqueName: \"kubernetes.io/projected/52a838aa-0c34-4c19-8764-dd2054bc0b78-kube-api-access-g565g\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.642515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.642484 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j"] Nov 01 16:00:45.646455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.646434 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.652710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.652677 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"distinct-w6-43fac8-kserve-self-signed-certs\"" Nov 01 16:00:45.655363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.655337 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"openshift-service-ca.crt\"" Nov 01 16:00:45.655491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.655463 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"kube-root-ca.crt\"" Nov 01 16:00:45.657079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.657054 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j"] Nov 01 16:00:45.680238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.680167 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.680375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.680286 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.680375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.680332 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g565g\" (UniqueName: \"kubernetes.io/projected/52a838aa-0c34-4c19-8764-dd2054bc0b78-kube-api-access-g565g\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.680943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.680925 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.682514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.682494 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.687996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.687970 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g565g\" (UniqueName: \"kubernetes.io/projected/52a838aa-0c34-4c19-8764-dd2054bc0b78-kube-api-access-g565g\") pod \"maas-api-revoke-sub-cbaaea25-2mcrd\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.689391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.689368 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:45.736626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.736604 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:45.753086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.753066 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-43fac8\"/\"default-dockercfg-z2zvx\"" Nov 01 16:00:45.784554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.784517 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813003eb-88ca-4744-919e-085b1ce97da7-tls-certs\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.784745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.784587 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqlpk\" (UniqueName: \"kubernetes.io/projected/813003eb-88ca-4744-919e-085b1ce97da7-kube-api-access-lqlpk\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.784745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.784643 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-home\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.784745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.784702 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-dshm\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.784922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.784741 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-model-cache\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.784922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.784813 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-tmp-dir\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.884245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.884212 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:45.885711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.885687 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-home\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.885810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.885791 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-dshm\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.885868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.885843 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-model-cache\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.885923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.885878 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-tmp-dir\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.886059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.886043 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813003eb-88ca-4744-919e-085b1ce97da7-tls-certs\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.886101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.886079 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-home\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.886144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.886098 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqlpk\" (UniqueName: \"kubernetes.io/projected/813003eb-88ca-4744-919e-085b1ce97da7-kube-api-access-lqlpk\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.886466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.886351 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-model-cache\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.886466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.886425 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-tmp-dir\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.888838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.888817 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813003eb-88ca-4744-919e-085b1ce97da7-tls-certs\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.890994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.890974 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-dshm\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.894187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.894163 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqlpk\" (UniqueName: \"kubernetes.io/projected/813003eb-88ca-4744-919e-085b1ce97da7-kube-api-access-lqlpk\") pod \"distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:45.964022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:45.963942 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:46.046793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.045803 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd"] Nov 01 16:00:46.049928 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:46.049901 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52a838aa_0c34_4c19_8764_dd2054bc0b78.slice/crio-5ebcb74ec17fdf9d55279997423aa4b97390594abb4173fd28fe25f75b4a07e5 WatchSource:0}: Error finding container 5ebcb74ec17fdf9d55279997423aa4b97390594abb4173fd28fe25f75b4a07e5: Status 404 returned error can't find the container with id 5ebcb74ec17fdf9d55279997423aa4b97390594abb4173fd28fe25f75b4a07e5 Nov 01 16:00:46.256494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.256455 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j"] Nov 01 16:00:46.257452 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:46.257426 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod813003eb_88ca_4744_919e_085b1ce97da7.slice/crio-f389ffef76db0b90b5e0fc86b5f1886bf3c141b0a4059b40429663a547e9326a WatchSource:0}: Error finding container f389ffef76db0b90b5e0fc86b5f1886bf3c141b0a4059b40429663a547e9326a: Status 404 returned error can't find the container with id f389ffef76db0b90b5e0fc86b5f1886bf3c141b0a4059b40429663a547e9326a Nov 01 16:00:46.286657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.286632 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t"] Nov 01 16:00:46.291111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.291093 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.298534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.298510 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t"] Nov 01 16:00:46.391815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.391789 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23895580-3977-4307-a7d5-66cf8f425666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.391956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.391936 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j76xf\" (UniqueName: \"kubernetes.io/projected/23895580-3977-4307-a7d5-66cf8f425666-kube-api-access-j76xf\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.392060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.392043 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23895580-3977-4307-a7d5-66cf8f425666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.492683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.492648 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j76xf\" (UniqueName: \"kubernetes.io/projected/23895580-3977-4307-a7d5-66cf8f425666-kube-api-access-j76xf\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.493201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.492719 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23895580-3977-4307-a7d5-66cf8f425666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.493201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.492845 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23895580-3977-4307-a7d5-66cf8f425666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.493490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.493466 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23895580-3977-4307-a7d5-66cf8f425666-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.495256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.495233 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23895580-3977-4307-a7d5-66cf8f425666-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.500338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.500316 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j76xf\" (UniqueName: \"kubernetes.io/projected/23895580-3977-4307-a7d5-66cf8f425666-kube-api-access-j76xf\") pod \"maas-api-revoke-sub-e113b817-7994t\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.610939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.610914 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:46.783484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.783450 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:46.889072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.889046 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t"] Nov 01 16:00:46.890052 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:46.890024 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23895580_3977_4307_a7d5_66cf8f425666.slice/crio-0efd049bc895b384a1fb9c3d9b9796695838e644067e0fd838a625a5e2e29ab2 WatchSource:0}: Error finding container 0efd049bc895b384a1fb9c3d9b9796695838e644067e0fd838a625a5e2e29ab2: Status 404 returned error can't find the container with id 0efd049bc895b384a1fb9c3d9b9796695838e644067e0fd838a625a5e2e29ab2 Nov 01 16:00:46.893219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.893188 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" event={"ID":"52a838aa-0c34-4c19-8764-dd2054bc0b78","Type":"ContainerStarted","Data":"f217e16b6ba8f8c9e9d0a02dc8210167ee4fdd2c3de9208804fdb9a8565d5411"} Nov 01 16:00:46.893343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.893247 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" event={"ID":"52a838aa-0c34-4c19-8764-dd2054bc0b78","Type":"ContainerStarted","Data":"5ebcb74ec17fdf9d55279997423aa4b97390594abb4173fd28fe25f75b4a07e5"} Nov 01 16:00:46.895139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.895109 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" event={"ID":"813003eb-88ca-4744-919e-085b1ce97da7","Type":"ContainerStarted","Data":"76c42123c400cb65c94a7bf4bf7a6b564b12a2244af65733f185c4dd37cef467"} Nov 01 16:00:46.895238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.895153 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" event={"ID":"813003eb-88ca-4744-919e-085b1ce97da7","Type":"ContainerStarted","Data":"f389ffef76db0b90b5e0fc86b5f1886bf3c141b0a4059b40429663a547e9326a"} Nov 01 16:00:46.915208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.915152 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" podStartSLOduration=1.915138607 podStartE2EDuration="1.915138607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:46.909859776 +0000 UTC m=+784.613025709" watchObservedRunningTime="2026-11-01 16:00:46.915138607 +0000 UTC m=+784.618304544" Nov 01 16:00:46.927218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.927179 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podStartSLOduration=1.92716711 podStartE2EDuration="1.92716711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:00:46.925460871 +0000 UTC m=+784.628626807" watchObservedRunningTime="2026-11-01 16:00:46.92716711 +0000 UTC m=+784.630333046" Nov 01 16:00:46.964164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:46.964137 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:47.557787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:47.557683 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:47.903295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:47.903264 4573 generic.go:356] "Generic (PLEG): container finished" podID="23895580-3977-4307-a7d5-66cf8f425666" containerID="f5cc9b4c424a0dc6cfee892ca5611f85fa369eabd2021fb0a2ca81e9b1084eb5" exitCode=0 Nov 01 16:00:47.903430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:47.903341 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" event={"ID":"23895580-3977-4307-a7d5-66cf8f425666","Type":"ContainerDied","Data":"f5cc9b4c424a0dc6cfee892ca5611f85fa369eabd2021fb0a2ca81e9b1084eb5"} Nov 01 16:00:47.903430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:47.903390 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" event={"ID":"23895580-3977-4307-a7d5-66cf8f425666","Type":"ContainerStarted","Data":"0efd049bc895b384a1fb9c3d9b9796695838e644067e0fd838a625a5e2e29ab2"} Nov 01 16:00:47.905021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:47.905002 4573 generic.go:356] "Generic (PLEG): container finished" podID="52a838aa-0c34-4c19-8764-dd2054bc0b78" containerID="f217e16b6ba8f8c9e9d0a02dc8210167ee4fdd2c3de9208804fdb9a8565d5411" exitCode=0 Nov 01 16:00:47.905129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:47.905087 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" event={"ID":"52a838aa-0c34-4c19-8764-dd2054bc0b78","Type":"ContainerDied","Data":"f217e16b6ba8f8c9e9d0a02dc8210167ee4fdd2c3de9208804fdb9a8565d5411"} Nov 01 16:00:48.344613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:48.344530 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:49.063333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.063308 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:49.089805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.089780 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:49.129506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.129473 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23895580-3977-4307-a7d5-66cf8f425666-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23895580-3977-4307-a7d5-66cf8f425666-maas-api-xks-ca\") pod \"23895580-3977-4307-a7d5-66cf8f425666\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " Nov 01 16:00:49.129658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.129545 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52a838aa-0c34-4c19-8764-dd2054bc0b78-kube-api-access-g565g\" (UniqueName: \"kubernetes.io/projected/52a838aa-0c34-4c19-8764-dd2054bc0b78-kube-api-access-g565g\") pod \"52a838aa-0c34-4c19-8764-dd2054bc0b78\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " Nov 01 16:00:49.129658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.129642 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23895580-3977-4307-a7d5-66cf8f425666-kube-api-access-j76xf\" (UniqueName: \"kubernetes.io/projected/23895580-3977-4307-a7d5-66cf8f425666-kube-api-access-j76xf\") pod \"23895580-3977-4307-a7d5-66cf8f425666\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " Nov 01 16:00:49.129788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.129718 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23895580-3977-4307-a7d5-66cf8f425666-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23895580-3977-4307-a7d5-66cf8f425666-maas-api-openshift-ca\") pod \"23895580-3977-4307-a7d5-66cf8f425666\" (UID: \"23895580-3977-4307-a7d5-66cf8f425666\") " Nov 01 16:00:49.129886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.129849 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-xks-ca\") pod \"52a838aa-0c34-4c19-8764-dd2054bc0b78\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " Nov 01 16:00:49.129955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.129897 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-openshift-ca\") pod \"52a838aa-0c34-4c19-8764-dd2054bc0b78\" (UID: \"52a838aa-0c34-4c19-8764-dd2054bc0b78\") " Nov 01 16:00:49.130491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.130216 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23895580-3977-4307-a7d5-66cf8f425666-maas-api-openshift-ca" pod "23895580-3977-4307-a7d5-66cf8f425666" (UID: "23895580-3977-4307-a7d5-66cf8f425666"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:49.130737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.130705 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23895580-3977-4307-a7d5-66cf8f425666-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:49.130921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.130896 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-openshift-ca" pod "52a838aa-0c34-4c19-8764-dd2054bc0b78" (UID: "52a838aa-0c34-4c19-8764-dd2054bc0b78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:00:49.132853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.132305 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23895580-3977-4307-a7d5-66cf8f425666-kube-api-access-j76xf" pod "23895580-3977-4307-a7d5-66cf8f425666" (UID: "23895580-3977-4307-a7d5-66cf8f425666"). InnerVolumeSpecName "kube-api-access-j76xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:49.132853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.132341 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23895580-3977-4307-a7d5-66cf8f425666-maas-api-xks-ca" pod "23895580-3977-4307-a7d5-66cf8f425666" (UID: "23895580-3977-4307-a7d5-66cf8f425666"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:49.132853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.132381 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52a838aa-0c34-4c19-8764-dd2054bc0b78-kube-api-access-g565g" pod "52a838aa-0c34-4c19-8764-dd2054bc0b78" (UID: "52a838aa-0c34-4c19-8764-dd2054bc0b78"). InnerVolumeSpecName "kube-api-access-g565g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:00:49.132853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.132842 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-xks-ca" pod "52a838aa-0c34-4c19-8764-dd2054bc0b78" (UID: "52a838aa-0c34-4c19-8764-dd2054bc0b78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:00:49.232197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.232127 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:49.232197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.232156 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52a838aa-0c34-4c19-8764-dd2054bc0b78-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:49.232197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.232168 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23895580-3977-4307-a7d5-66cf8f425666-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:49.232197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.232178 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g565g\" (UniqueName: \"kubernetes.io/projected/52a838aa-0c34-4c19-8764-dd2054bc0b78-kube-api-access-g565g\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:49.232197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.232187 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j76xf\" (UniqueName: \"kubernetes.io/projected/23895580-3977-4307-a7d5-66cf8f425666-kube-api-access-j76xf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:00:49.262842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.262806 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:49.919361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.919322 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" Nov 01 16:00:49.919361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.919352 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd" event={"ID":"52a838aa-0c34-4c19-8764-dd2054bc0b78","Type":"ContainerDied","Data":"5ebcb74ec17fdf9d55279997423aa4b97390594abb4173fd28fe25f75b4a07e5"} Nov 01 16:00:49.919623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.919422 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ebcb74ec17fdf9d55279997423aa4b97390594abb4173fd28fe25f75b4a07e5" Nov 01 16:00:49.920801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.920775 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" event={"ID":"23895580-3977-4307-a7d5-66cf8f425666","Type":"ContainerDied","Data":"0efd049bc895b384a1fb9c3d9b9796695838e644067e0fd838a625a5e2e29ab2"} Nov 01 16:00:49.920801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.920800 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t" Nov 01 16:00:49.921018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:49.920815 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0efd049bc895b384a1fb9c3d9b9796695838e644067e0fd838a625a5e2e29ab2" Nov 01 16:00:50.042891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:50.042861 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:50.839441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:50.839403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:51.057518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:51.057480 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:51.669276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:51.669247 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:52.362035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:52.361979 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:52.966636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:52.966608 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:53.061206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:53.061177 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:53.864331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:53.864284 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:54.363520 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:00:54.363489 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:00:54.577459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:54.577422 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:55.333662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:55.333630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:55.964075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:55.964044 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:55.970362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:55.970337 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:55.977294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:55.977274 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:56.559648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:56.559607 4573 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","podc08b4081-5132-4dc8-95b8-df8769d19e57"] err="unable to destroy cgroup paths for cgroup [kubepods burstable podc08b4081-5132-4dc8-95b8-df8769d19e57] : Timed out while waiting for systemd to remove kubepods-burstable-podc08b4081_5132_4dc8_95b8_df8769d19e57.slice" Nov 01 16:00:56.559648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:56.559642 4573 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="c08b4081-5132-4dc8-95b8-df8769d19e57" err="failed to delete cgroup paths for [kubepods burstable podc08b4081-5132-4dc8-95b8-df8769d19e57] : unable to destroy cgroup paths for cgroup [kubepods burstable podc08b4081-5132-4dc8-95b8-df8769d19e57] : Timed out while waiting for systemd to remove kubepods-burstable-podc08b4081_5132_4dc8_95b8_df8769d19e57.slice" Nov 01 16:00:56.681712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:56.681682 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:56.858746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:56.858710 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:00:56.976278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:56.976250 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:00:57.565876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:57.565842 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:58.360041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.360006 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w"] Nov 01 16:00:58.361112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.361085 4573 cpu_manager.go:414] "containerMap: removing container" podUID="52a838aa-0c34-4c19-8764-dd2054bc0b78" containerName="revoke-keys" Nov 01 16:00:58.361268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.361254 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52a838aa-0c34-4c19-8764-dd2054bc0b78" containerName="revoke-keys" Nov 01 16:00:58.361414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.361401 4573 cpu_manager.go:414] "containerMap: removing container" podUID="23895580-3977-4307-a7d5-66cf8f425666" containerName="revoke-keys" Nov 01 16:00:58.361517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.361505 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23895580-3977-4307-a7d5-66cf8f425666" containerName="revoke-keys" Nov 01 16:00:58.361787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.361769 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="52a838aa-0c34-4c19-8764-dd2054bc0b78" containerName="revoke-keys" Nov 01 16:00:58.361922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.361908 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="23895580-3977-4307-a7d5-66cf8f425666" containerName="revoke-keys" Nov 01 16:00:58.377271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.377247 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.471368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.471317 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.471640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.471617 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.471798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.471779 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.472016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.471999 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.472170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.472151 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.472380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.472360 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kppw\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-kube-api-access-2kppw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.472514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.472498 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.472625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.472609 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.472724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.472711 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3909dc-41fc-4299-ad14-5a36d69318d8-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.612816 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w"] Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.613652 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kppw\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-kube-api-access-2kppw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.613705 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.613731 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.613772 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3909dc-41fc-4299-ad14-5a36d69318d8-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.613840 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.613893 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.613920 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.614096 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.614143 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.615616 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.645824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.616509 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3909dc-41fc-4299-ad14-5a36d69318d8-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.670775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.666081 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.671003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.669629 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kppw\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-kube-api-access-2kppw\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.671129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.669926 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.678770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.675973 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.684227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.684201 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.741983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.741955 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.744267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.744208 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:58.779341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:58.779313 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:00:59.157150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.157115 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:00:59.223867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.223736 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs"] Nov 01 16:00:59.232792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.232764 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs"] Nov 01 16:00:59.233838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.233237 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.318841 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.318886 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2kc5\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-kube-api-access-t2kc5\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.318909 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.318924 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.318944 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.318960 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.319004 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.319025 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.319488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.319086 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.362824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.362796 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w"] Nov 01 16:00:59.363425 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:59.363393 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3909dc_41fc_4299_ad14_5a36d69318d8.slice/crio-22f3488eaac43935680f73dc4ec237386e2a74198ba3b9fe1435accb2dd5bdf3 WatchSource:0}: Error finding container 22f3488eaac43935680f73dc4ec237386e2a74198ba3b9fe1435accb2dd5bdf3: Status 404 returned error can't find the container with id 22f3488eaac43935680f73dc4ec237386e2a74198ba3b9fe1435accb2dd5bdf3 Nov 01 16:00:59.366695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.366662 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:59.366806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.366728 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:59.366806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.366773 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:59.419767 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.419722 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.419898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.419792 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2kc5\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-kube-api-access-t2kc5\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.419898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.419824 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.419898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.419848 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.419898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.419878 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.420133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.419904 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.420133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.419960 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.420133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.419985 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.420133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.420052 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.420332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.420126 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.420690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.420466 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.420690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.420642 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.420878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.420713 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.421215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.421188 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.423035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.423009 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.423401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.423381 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.430644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.430619 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.430727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.430692 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2kc5\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-kube-api-access-t2kc5\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.558530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.558438 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:00:59.847038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.847005 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs"] Nov 01 16:00:59.847905 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:00:59.847879 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bf3f4c_76f8_4434_8015_b6515fd018d3.slice/crio-504d5716b36c5cecb92822154b2e508480120a49351bfb14c9c2d977f37076f4 WatchSource:0}: Error finding container 504d5716b36c5cecb92822154b2e508480120a49351bfb14c9c2d977f37076f4: Status 404 returned error can't find the container with id 504d5716b36c5cecb92822154b2e508480120a49351bfb14c9c2d977f37076f4 Nov 01 16:00:59.850371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.850319 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:59.850486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.850412 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:00:59.850486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:00:59.850456 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:00.008023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.007978 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" event={"ID":"9e3909dc-41fc-4299-ad14-5a36d69318d8","Type":"ContainerStarted","Data":"41fed38f636e146318bca59309a081783923e74dabb273585798877717098fe8"} Nov 01 16:01:00.008196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.008043 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" event={"ID":"9e3909dc-41fc-4299-ad14-5a36d69318d8","Type":"ContainerStarted","Data":"22f3488eaac43935680f73dc4ec237386e2a74198ba3b9fe1435accb2dd5bdf3"} Nov 01 16:01:00.010057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.010024 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" event={"ID":"f4bf3f4c-76f8-4434-8015-b6515fd018d3","Type":"ContainerStarted","Data":"b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9"} Nov 01 16:01:00.010193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.010069 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" event={"ID":"f4bf3f4c-76f8-4434-8015-b6515fd018d3","Type":"ContainerStarted","Data":"504d5716b36c5cecb92822154b2e508480120a49351bfb14c9c2d977f37076f4"} Nov 01 16:01:00.056342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.056282 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podStartSLOduration=2.056266917 podStartE2EDuration="2.056266917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:00.05503902 +0000 UTC m=+797.758204959" watchObservedRunningTime="2026-11-01 16:01:00.056266917 +0000 UTC m=+797.759432852" Nov 01 16:01:00.083943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.083890 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podStartSLOduration=1.083875647 podStartE2EDuration="1.083875647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:00:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:00.080574139 +0000 UTC m=+797.783740077" watchObservedRunningTime="2026-11-01 16:01:00.083875647 +0000 UTC m=+797.787041584" Nov 01 16:01:00.119065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.119028 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:00.559707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.559629 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:01:00.561241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.561216 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 16:01:00.561350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.561263 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 16:01:00.786855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.786816 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:01:00.789085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.789051 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:01:00.789216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:00.789110 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 16:01:01.007773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.006837 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:01.437257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.437179 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56"] Nov 01 16:01:01.441602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.441582 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.447019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.446787 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"kube-root-ca.crt\"" Nov 01 16:01:01.449222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.449193 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"openshift-service-ca.crt\"" Nov 01 16:01:01.450552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.450471 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"default-dockercfg-4dxd5\"" Nov 01 16:01:01.450655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.450603 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-dfff59\"/\"distinct-w1-dfff59-kserve-self-signed-certs\"" Nov 01 16:01:01.453796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.453074 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56"] Nov 01 16:01:01.519819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.519453 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:01.550793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.550731 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-model-cache\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.550963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.550850 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-home\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.550963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.550919 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af872790-62d6-41e2-be43-07c7511b047b-tls-certs\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.551046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.551008 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5kdf\" (UniqueName: \"kubernetes.io/projected/af872790-62d6-41e2-be43-07c7511b047b-kube-api-access-t5kdf\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.551156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.551144 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-tmp-dir\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.551189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.551168 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-dshm\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.559846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.559824 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 16:01:01.559900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.559884 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 16:01:01.660921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.656830 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-model-cache\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.660921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.657688 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-home\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.660921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.657808 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af872790-62d6-41e2-be43-07c7511b047b-tls-certs\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.660921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.657931 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5kdf\" (UniqueName: \"kubernetes.io/projected/af872790-62d6-41e2-be43-07c7511b047b-kube-api-access-t5kdf\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.660921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.658180 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-tmp-dir\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.660921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.658218 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-dshm\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.660921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.657288 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-model-cache\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.671774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.662067 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-home\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.680636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.680561 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-tmp-dir\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.691175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.691056 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-dshm\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.698773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.694963 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5kdf\" (UniqueName: \"kubernetes.io/projected/af872790-62d6-41e2-be43-07c7511b047b-kube-api-access-t5kdf\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.698773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.696395 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af872790-62d6-41e2-be43-07c7511b047b-tls-certs\") pod \"distinct-w1-dfff59-kserve-56fccf5f84-cts56\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.756393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.756358 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:01.793778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.780350 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:01:01.793778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:01.780427 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 16:01:02.130726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:02.130690 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:02.154522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:02.154499 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56"] Nov 01 16:01:02.559738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:02.559644 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 16:01:02.559738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:02.559693 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 16:01:02.783777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:02.783095 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:01:02.783777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:02.783155 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 16:01:03.027632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.027576 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:03.038887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.038857 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" event={"ID":"af872790-62d6-41e2-be43-07c7511b047b","Type":"ContainerStarted","Data":"095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e"} Nov 01 16:01:03.039057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.038895 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" event={"ID":"af872790-62d6-41e2-be43-07c7511b047b","Type":"ContainerStarted","Data":"39138a47ff0af2806ea31d3f9d0a6705b10deaa9373bc0904d68eefb2f75b9ab"} Nov 01 16:01:03.056581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.056535 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podStartSLOduration=2.056522024 podStartE2EDuration="2.056522024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:03.053480108 +0000 UTC m=+800.756646044" watchObservedRunningTime="2026-11-01 16:01:03.056522024 +0000 UTC m=+800.759687961" Nov 01 16:01:03.559450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.559411 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 16:01:03.559836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.559467 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 16:01:03.756518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.756479 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:03.781783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.780263 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:01:03.781783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.780321 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 16:01:03.929659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:03.929615 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:04.559496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:04.559460 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 16:01:04.559921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:04.559514 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 16:01:04.803474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:04.803423 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:01:04.803825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:04.803794 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 16:01:04.959958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:04.959924 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:05.559459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:05.559428 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 16:01:05.559625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:05.559476 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 16:01:05.781113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:05.781076 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 16:01:05.781399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:05.781365 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 16:01:05.865828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:05.862164 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:06.559625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:06.559586 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 01 16:01:06.560052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:06.559656 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 01 16:01:06.608726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:06.608684 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:06.845303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:06.845206 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:01:06.845789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:06.845744 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:01:06.846558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:06.846526 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:01:07.425376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:07.425341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:07.563319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:07.563289 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:01:07.563810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:07.563780 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:01:07.564597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:07.564574 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:01:08.401669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:08.401636 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:08.491279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:08.491245 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:09.070019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:09.069987 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:09.754061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:09.754028 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:10.075192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:10.075115 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:10.472547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:10.472515 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:11.049038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.049003 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg"] Nov 01 16:01:11.055464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.055436 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.060563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.060453 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"openshift-service-ca.crt\"" Nov 01 16:01:11.060563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.060501 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"kube-root-ca.crt\"" Nov 01 16:01:11.060563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.060517 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"default-dockercfg-n49qh\"" Nov 01 16:01:11.060877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.060596 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9f4dc9\"/\"distinct-2-w4-9f4dc9-kserve-self-signed-certs\"" Nov 01 16:01:11.062986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.062962 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg"] Nov 01 16:01:11.083108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.083062 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:11.141316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.141286 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-model-cache\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.141463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.141428 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-home\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.141513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.141497 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-tmp-dir\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.141550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.141522 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6b94\" (UniqueName: \"kubernetes.io/projected/839701ef-516c-4451-b096-50ca25cdcabc-kube-api-access-h6b94\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.141621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.141607 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-dshm\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.141682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.141670 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/839701ef-516c-4451-b096-50ca25cdcabc-tls-certs\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.243393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.243362 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/839701ef-516c-4451-b096-50ca25cdcabc-tls-certs\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.243552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.243444 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-model-cache\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.243552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.243538 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-home\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.243624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.243563 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-tmp-dir\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.243624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.243578 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6b94\" (UniqueName: \"kubernetes.io/projected/839701ef-516c-4451-b096-50ca25cdcabc-kube-api-access-h6b94\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.243708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.243620 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-dshm\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.243997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.243974 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-model-cache\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.244114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.244005 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-home\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.244114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.244029 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-tmp-dir\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.245858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.245825 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-dshm\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.246171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.246152 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/839701ef-516c-4451-b096-50ca25cdcabc-tls-certs\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.250935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.250914 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6b94\" (UniqueName: \"kubernetes.io/projected/839701ef-516c-4451-b096-50ca25cdcabc-kube-api-access-h6b94\") pod \"distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.254274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.254254 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:11.372562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.372534 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:11.663065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.663039 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg"] Nov 01 16:01:11.663673 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:01:11.663650 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod839701ef_516c_4451_b096_50ca25cdcabc.slice/crio-e079d483ea5570569ae126c811a5d02763a0ef61043d088014ad0ddd21e4544e WatchSource:0}: Error finding container e079d483ea5570569ae126c811a5d02763a0ef61043d088014ad0ddd21e4544e: Status 404 returned error can't find the container with id e079d483ea5570569ae126c811a5d02763a0ef61043d088014ad0ddd21e4544e Nov 01 16:01:11.756896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.756864 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:11.770875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.770851 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:11.959722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:11.959642 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:12.108610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.108573 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" event={"ID":"839701ef-516c-4451-b096-50ca25cdcabc","Type":"ContainerStarted","Data":"8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580"} Nov 01 16:01:12.108610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.108620 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" event={"ID":"839701ef-516c-4451-b096-50ca25cdcabc","Type":"ContainerStarted","Data":"e079d483ea5570569ae126c811a5d02763a0ef61043d088014ad0ddd21e4544e"} Nov 01 16:01:12.119796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.119772 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:12.128124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.128081 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podStartSLOduration=1.12806919 podStartE2EDuration="1.12806919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:12.124001535 +0000 UTC m=+809.827167474" watchObservedRunningTime="2026-11-01 16:01:12.12806919 +0000 UTC m=+809.831235126" Nov 01 16:01:12.353979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.353885 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m"] Nov 01 16:01:12.358362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.358335 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.362556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.362529 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 16:01:12.362724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.362710 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 16:01:12.362867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.362704 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-k9xjt\"" Nov 01 16:01:12.362929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.362894 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 16:01:12.374299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.374271 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:12.375694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.375671 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m"] Nov 01 16:01:12.460509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.460463 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjlhf\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-kube-api-access-fjlhf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.460697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.460520 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.460697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.460620 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.460697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.460658 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.460891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.460866 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.460938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.460928 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.460973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.460947 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.561694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.561662 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjlhf\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-kube-api-access-fjlhf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.561875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.561707 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.561875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.561746 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.561956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.561898 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.562209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.562185 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.562318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.562235 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.562318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.562280 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.563195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.563172 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.564401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.564346 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.564708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.564685 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.564817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.564790 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.565042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.565022 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.569465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.569443 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjlhf\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-kube-api-access-fjlhf\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.569786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.569740 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-27n7m\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.677881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.677857 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:12.743087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.743023 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:12.965600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:12.965570 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m"] Nov 01 16:01:12.965793 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:01:12.965770 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55ba933_be66_4aaa_a2a7_6b98b9e000e0.slice/crio-68741f58b96281d03c7f98c1fb24c72d1d610766f998ca5bb7a380971094490c WatchSource:0}: Error finding container 68741f58b96281d03c7f98c1fb24c72d1d610766f998ca5bb7a380971094490c: Status 404 returned error can't find the container with id 68741f58b96281d03c7f98c1fb24c72d1d610766f998ca5bb7a380971094490c Nov 01 16:01:13.117897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.117854 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" event={"ID":"a55ba933-be66-4aaa-a2a7-6b98b9e000e0","Type":"ContainerStarted","Data":"68741f58b96281d03c7f98c1fb24c72d1d610766f998ca5bb7a380971094490c"} Nov 01 16:01:13.488416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.488382 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:13.557096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.557063 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j"] Nov 01 16:01:13.562707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.562677 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.579004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.578973 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j"] Nov 01 16:01:13.636699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.636663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:13.679249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679213 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.679426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679324 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.679426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679357 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.679426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679373 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.679580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679499 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.679580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679544 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.679692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679657 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vlfw\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-kube-api-access-9vlfw\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.679767 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679697 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.679829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.679783 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.705106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.705075 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f"] Nov 01 16:01:13.710362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.710336 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.726214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.726178 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f"] Nov 01 16:01:13.781147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781066 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlfw\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-kube-api-access-9vlfw\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781124 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781165 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781231 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781297 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781413 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781446 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781518 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781580 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781659 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781676 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.781746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.781718 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.782043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.782022 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.782043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.782036 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.783883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.783864 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.784141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.784123 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.788925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.788901 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.789072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.789058 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlfw\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-kube-api-access-9vlfw\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.883433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883387 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.883639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883503 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.883639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883586 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.883639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883614 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.883639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883637 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.883862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883660 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.883862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883740 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.883862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883799 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.883862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.883853 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4blpj\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-kube-api-access-4blpj\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.887937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.887233 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:13.984992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.984954 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985043 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985115 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985155 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985184 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985215 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985244 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985287 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985381 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4blpj\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-kube-api-access-4blpj\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985520 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.985638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985618 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.986090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.985909 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.986396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.986351 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.986706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.986687 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.988671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.988648 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.988933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.988915 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.994722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.994695 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:13.995093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:13.995072 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4blpj\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-kube-api-access-4blpj\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:14.040991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.040919 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:14.175101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.175069 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j"] Nov 01 16:01:14.175498 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:01:14.175153 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf3d53a_d18e_48b9_ae20_21341d5f6fed.slice/crio-8989d8536e46c64102256bbd846683daa455d6b288d9bad9e66eddb3b47dd7e0 WatchSource:0}: Error finding container 8989d8536e46c64102256bbd846683daa455d6b288d9bad9e66eddb3b47dd7e0: Status 404 returned error can't find the container with id 8989d8536e46c64102256bbd846683daa455d6b288d9bad9e66eddb3b47dd7e0 Nov 01 16:01:14.178486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.178446 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.178609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.178537 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.178609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.178585 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.362741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.362708 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f"] Nov 01 16:01:14.363275 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:01:14.363248 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f48c1b6_aaf2_4b54_a5d8_35d3b6b6a6a8.slice/crio-8aebe1f906861b96d011c661dd3753cd917ce0f0c1876849e745ff25fcccb654 WatchSource:0}: Error finding container 8aebe1f906861b96d011c661dd3753cd917ce0f0c1876849e745ff25fcccb654: Status 404 returned error can't find the container with id 8aebe1f906861b96d011c661dd3753cd917ce0f0c1876849e745ff25fcccb654 Nov 01 16:01:14.366366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.366332 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.366465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.366399 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.366465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.366431 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:14.593117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:14.593080 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:15.143692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.142927 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" event={"ID":"aaf3d53a-d18e-48b9-ae20-21341d5f6fed","Type":"ContainerStarted","Data":"15b8aa6288d5971648ab6642e23e70ada85e1d13f797115d435730f013a51eaf"} Nov 01 16:01:15.143692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.143005 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" event={"ID":"aaf3d53a-d18e-48b9-ae20-21341d5f6fed","Type":"ContainerStarted","Data":"8989d8536e46c64102256bbd846683daa455d6b288d9bad9e66eddb3b47dd7e0"} Nov 01 16:01:15.145489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.145461 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" event={"ID":"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8","Type":"ContainerStarted","Data":"a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4"} Nov 01 16:01:15.145602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.145510 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" event={"ID":"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8","Type":"ContainerStarted","Data":"8aebe1f906861b96d011c661dd3753cd917ce0f0c1876849e745ff25fcccb654"} Nov 01 16:01:15.191779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.186889 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podStartSLOduration=2.186868792 podStartE2EDuration="2.186868792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:15.161819917 +0000 UTC m=+812.864985853" watchObservedRunningTime="2026-11-01 16:01:15.186868792 +0000 UTC m=+812.890034730" Nov 01 16:01:15.211931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.211878 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" podStartSLOduration=2.211859138 podStartE2EDuration="2.211859138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:15.208474222 +0000 UTC m=+812.911640158" watchObservedRunningTime="2026-11-01 16:01:15.211859138 +0000 UTC m=+812.915025080" Nov 01 16:01:15.634383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.633922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:15.905170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.904131 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:15.911350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.910933 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 16:01:15.911350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:15.911002 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 16:01:16.041888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:16.041853 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:16.043970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:16.043945 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:16.044109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:16.043993 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 16:01:16.605809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:16.603155 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:16.896774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:16.896722 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 16:01:16.897060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:16.897030 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 16:01:17.041969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:17.041936 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:17.042130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:17.041989 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 16:01:17.723398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:17.723303 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:17.888775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:17.887900 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 16:01:17.888775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:17.887955 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 16:01:18.041854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:18.041766 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:18.041854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:18.041813 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 16:01:18.256542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:18.256508 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:18.256610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:18.256591 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:18.901804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:18.897783 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 16:01:18.901804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:18.897856 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 16:01:18.913158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:18.904263 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:19.048982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.048347 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:19.048982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.048425 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 16:01:19.251216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.249192 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" event={"ID":"a55ba933-be66-4aaa-a2a7-6b98b9e000e0","Type":"ContainerStarted","Data":"46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4"} Nov 01 16:01:19.251216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.249356 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:19.251216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.251008 4573 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-27n7m container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:01:19.251216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.251055 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" podUID="a55ba933-be66-4aaa-a2a7-6b98b9e000e0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:01:19.312776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.293702 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" podStartSLOduration=2.006374246 podStartE2EDuration="7.293682141s" totalImagesPullingTime="5.287307895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:19.270158424 +0000 UTC m=+816.973324373" watchObservedRunningTime="2026-11-01 16:01:19.293682141 +0000 UTC m=+816.996848081" Nov 01 16:01:19.342360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.342312 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:19.441423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.441373 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:19.891775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.890198 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 16:01:19.891775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:19.890261 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 16:01:20.041544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:20.041513 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:20.042017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:20.041567 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 16:01:20.202837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:20.201611 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:20.257021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:20.256994 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:01:20.887984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:20.887940 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 01 16:01:20.888196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:20.888006 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 01 16:01:21.042095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:21.042061 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 01 16:01:21.042623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:21.042114 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 01 16:01:21.320151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:21.320054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:21.382778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:21.376083 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:21.399161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:21.399128 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:21.917139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:21.917115 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:21.918832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:21.918807 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:21.919662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:21.919648 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:01:22.122078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:22.122047 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:22.124145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:22.124110 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:22.148014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:22.147985 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:01:22.399117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:22.399085 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:23.558882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:23.558829 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.559484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:23.559456 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:01:23.834834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:23.813827 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:01:23.834834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:23.814292 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:01:23.834834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:23.814601 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:01:23.834834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:23.831866 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:23.890939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:23.890889 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.020054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:24.020004 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.135139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:24.135090 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.407051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:24.406948 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.572000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:24.557794 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:24.572482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:24.572137 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:24.710536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:24.710073 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:01:24.735446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:24.733674 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:01:24.769709 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:01:24.769628 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:01:26.516701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:26.516664 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:27.362637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:27.362605 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:28.150732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:28.150697 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:28.669240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:28.669207 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:29.156571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:29.156542 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:30.071643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:30.071607 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:30.281416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:30.281385 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:30.875523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:30.875488 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:31.824606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:31.824572 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:32.568481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:32.568449 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:33.266315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:33.266283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:33.960128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:33.960094 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:34.752603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:34.752573 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:35.382471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:35.382429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:36.053999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:36.053964 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:36.545866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:36.545840 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:36.857169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:36.857141 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:37.563293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:37.563257 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:38.359484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:38.359455 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:38.456458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:38.456425 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:39.169842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:39.169802 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:40.046278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:40.046239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:40.761779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:40.761710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:41.451578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:41.451548 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:42.070418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:42.070386 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:42.704640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:42.704605 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:43.391524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.391486 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:43.890030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.889995 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq"] Nov 01 16:01:43.896412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.896387 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.909250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.909228 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq"] Nov 01 16:01:43.987356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.987320 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.987537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.987365 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.987537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.987483 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.987537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.987516 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.987740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.987717 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.987855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.987800 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.987901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.987853 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.987960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.987943 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwj4\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-kube-api-access-zxwj4\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:43.988071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:43.988049 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.044618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.044579 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx"] Nov 01 16:01:44.050113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.050086 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.071467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.071440 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx"] Nov 01 16:01:44.089644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.089614 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.089849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.089666 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.089849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.089734 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.089849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.089773 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.089969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.089897 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.089969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.089934 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.090035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.089969 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.090035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.089999 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.090112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090033 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.090112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090062 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdzxc\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-kube-api-access-sdzxc\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.090218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090162 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.090283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090252 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c44a69a-e1e5-4767-8915-204e17dc3461-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.090341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090312 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.090396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090352 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.090463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090412 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.090515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090478 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwj4\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-kube-api-access-zxwj4\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.090570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090536 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.090623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.090573 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.091121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.091048 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.091121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.091086 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.091660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.091443 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.091660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.091618 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.091972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.091949 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.092551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.092531 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.092719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.092700 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.110452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.110422 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.111658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.111630 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwj4\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-kube-api-access-zxwj4\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.118568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.118549 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj"] Nov 01 16:01:44.124585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.124568 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.136494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.136475 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq"] Nov 01 16:01:44.142247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.142194 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj"] Nov 01 16:01:44.142342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.142276 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.158996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.158967 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:44.159309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.159288 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq"] Nov 01 16:01:44.192117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192086 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.192297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192131 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.192297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192164 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.192483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192443 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.192630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192593 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.192716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192659 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192723 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192799 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192839 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdzxc\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-kube-api-access-sdzxc\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192876 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192906 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192933 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192961 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.192992 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8d9\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-kube-api-access-zn8d9\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.193060 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.193090 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.193148 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.193177 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.193546 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.193704 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194050 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-certs\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194099 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194351 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-data\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194556 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194619 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194649 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-kube-api-access-btqfk\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194708 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194723 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194939 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-credential-socket\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.194971 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.195198 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c44a69a-e1e5-4767-8915-204e17dc3461-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.195876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.195873 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c44a69a-e1e5-4767-8915-204e17dc3461-istiod-ca-cert\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.197634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.197612 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-podinfo\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.199155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.199131 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-envoy\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.207980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.207954 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-token\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.208098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.208036 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdzxc\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-kube-api-access-sdzxc\") pod \"e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.214380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.214353 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:44.244546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.244513 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k"] Nov 01 16:01:44.249594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.249529 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.266939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.260657 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k"] Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.298874 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.298927 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.298970 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.298996 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299037 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299066 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299094 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299128 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8d9\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-kube-api-access-zn8d9\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299165 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pt42\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-kube-api-access-2pt42\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299212 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299242 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299288 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299323 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299349 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.299462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299378 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.300312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.300174 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.299419 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.300787 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-kube-api-access-btqfk\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.300840 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.300878 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.300905 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.300956 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301014 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301047 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301098 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301127 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301199 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301238 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301310 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301584 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-data\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.302046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.301584 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.302934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.302522 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.302934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.302791 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.303444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.303389 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istiod-ca-cert\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.303545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.303501 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.303873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.303732 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-credential-socket\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.303873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.303834 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-certs\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.306059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.306012 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.311247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.310832 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-podinfo\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.316892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.316519 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.324603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.324573 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.325247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.325222 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-kube-api-access-btqfk\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.325609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.325550 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-envoy\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.327322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.327294 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-token\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.328818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.328211 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8d9\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-kube-api-access-zn8d9\") pod \"e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.371163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.371132 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:44.402422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402341 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.402422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402406 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.402881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402511 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.402881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402573 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.402881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402605 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.402881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402647 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.402881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402745 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pt42\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-kube-api-access-2pt42\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.402881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402837 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.402881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.402873 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.403256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.403155 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.403256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.403233 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-data\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.404307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.403713 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-certs\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.404307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.403847 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-credential-socket\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.404307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.404257 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istiod-ca-cert\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.405625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.405604 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-podinfo\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.405790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.405744 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-envoy\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.413257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.413228 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-token\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.413383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.413366 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pt42\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-kube-api-access-2pt42\") pod \"e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.447450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.447198 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:44.464448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.464423 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:44.537923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.537811 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.538055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.537988 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.538130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.538097 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.562724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.561850 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq"] Nov 01 16:01:44.578809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.578783 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:44.671464 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:01:44.671264 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c44a69a_e1e5_4767_8915_204e17dc3461.slice/crio-9a8fc261632d6ba0037ddc90e35b107db62bf2582c548b9e8bacf52b678b94df WatchSource:0}: Error finding container 9a8fc261632d6ba0037ddc90e35b107db62bf2582c548b9e8bacf52b678b94df: Status 404 returned error can't find the container with id 9a8fc261632d6ba0037ddc90e35b107db62bf2582c548b9e8bacf52b678b94df Nov 01 16:01:44.673223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.672794 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx"] Nov 01 16:01:44.674434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.674393 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.674533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.674484 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.674599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.674559 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.773082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.769501 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq"] Nov 01 16:01:44.773942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.773922 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj"] Nov 01 16:01:44.784693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.784653 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.784826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.784741 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.784942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.784826 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.786653 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:01:44.786587 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc40bde30_47d6_4343_9ba4_780dab5f1ce8.slice/crio-981ac3db570f216ffe5935e0881402f1b00cd363d253a93f6eb39759a64b9035 WatchSource:0}: Error finding container 981ac3db570f216ffe5935e0881402f1b00cd363d253a93f6eb39759a64b9035: Status 404 returned error can't find the container with id 981ac3db570f216ffe5935e0881402f1b00cd363d253a93f6eb39759a64b9035 Nov 01 16:01:44.789796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.789730 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.789889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.789842 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.789958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.789883 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.874245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.874160 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" event={"ID":"c40bde30-47d6-4343-9ba4-780dab5f1ce8","Type":"ContainerStarted","Data":"981ac3db570f216ffe5935e0881402f1b00cd363d253a93f6eb39759a64b9035"} Nov 01 16:01:44.878007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.877127 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" event={"ID":"5c44a69a-e1e5-4767-8915-204e17dc3461","Type":"ContainerStarted","Data":"cd891c59a5cc83970989bfa71b9eced23ec07f6af41fdb7d912f9a46c6b151ec"} Nov 01 16:01:44.878007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.877167 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" event={"ID":"5c44a69a-e1e5-4767-8915-204e17dc3461","Type":"ContainerStarted","Data":"9a8fc261632d6ba0037ddc90e35b107db62bf2582c548b9e8bacf52b678b94df"} Nov 01 16:01:44.880906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.880880 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" event={"ID":"63c8337d-1c52-456e-b8e7-8afcd6f30eec","Type":"ContainerStarted","Data":"f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5"} Nov 01 16:01:44.880991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.880930 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" event={"ID":"63c8337d-1c52-456e-b8e7-8afcd6f30eec","Type":"ContainerStarted","Data":"cdec607e63f53adf366adabc709d777518a57cce1b9aadd39a2ffbe3880423a8"} Nov 01 16:01:44.883206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.883185 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" event={"ID":"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2","Type":"ContainerStarted","Data":"610ae9ca557ed456563810213fc88e2665675810fd6b67648b991eee3f54d137"} Nov 01 16:01:44.910671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.910584 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k"] Nov 01 16:01:44.917346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.917251 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podStartSLOduration=0.917231462 podStartE2EDuration="917.231462ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:44.908067523 +0000 UTC m=+842.611233460" watchObservedRunningTime="2026-11-01 16:01:44.917231462 +0000 UTC m=+842.620397399" Nov 01 16:01:44.921857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.921490 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.921857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.921575 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.921857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.921618 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:01:44.995050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:44.995007 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:45.216153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:45.216072 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:45.217242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:45.217217 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:45.217363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:45.217267 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:45.378573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:45.378544 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:45.378955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:45.378929 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:45.379127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:45.379104 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:45.955885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:45.955763 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" event={"ID":"0018bf27-76e3-49c8-bec5-05cadb0f4a9a","Type":"ContainerStarted","Data":"b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88"} Nov 01 16:01:45.955885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:45.955843 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" event={"ID":"0018bf27-76e3-49c8-bec5-05cadb0f4a9a","Type":"ContainerStarted","Data":"ec5c9ed48451784c3c65033c8cee6b489e8af54cc123a4d1f42e88ac76439d1b"} Nov 01 16:01:46.001773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.000711 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podStartSLOduration=3.000686138 podStartE2EDuration="3.000686138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:44.982609515 +0000 UTC m=+842.685775441" watchObservedRunningTime="2026-11-01 16:01:46.000686138 +0000 UTC m=+843.703852075" Nov 01 16:01:46.122045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.122006 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" event={"ID":"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2","Type":"ContainerStarted","Data":"04aad2811242bc854b88a1bc6824d45cf8b2e4acf142a27b13892cd67d913ed0"} Nov 01 16:01:46.126981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.126946 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" event={"ID":"c40bde30-47d6-4343-9ba4-780dab5f1ce8","Type":"ContainerStarted","Data":"bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9"} Nov 01 16:01:46.225839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.217587 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.225839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.217646 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:46.239775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.234162 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podStartSLOduration=2.234136116 podStartE2EDuration="2.234136116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:46.176112445 +0000 UTC m=+843.879278389" watchObservedRunningTime="2026-11-01 16:01:46.234136116 +0000 UTC m=+843.937302053" Nov 01 16:01:46.239775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.234337 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podStartSLOduration=2.234328077 podStartE2EDuration="2.234328077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:46.002386598 +0000 UTC m=+843.705552535" watchObservedRunningTime="2026-11-01 16:01:46.234328077 +0000 UTC m=+843.937494014" Nov 01 16:01:46.286066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.286004 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podStartSLOduration=2.285983795 podStartE2EDuration="2.285983795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:01:46.218562056 +0000 UTC m=+843.921728006" watchObservedRunningTime="2026-11-01 16:01:46.285983795 +0000 UTC m=+843.989149736" Nov 01 16:01:46.373269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.373229 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.373556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.373527 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:46.450118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.450081 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:46.450436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.450412 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.450611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.450587 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:46.470319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.469821 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.470319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.469887 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:46.531043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.530943 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:46.584951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.584920 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:46.585274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.585253 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:46.585419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.585397 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:46.599276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.599242 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:46.893777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.892488 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5"] Nov 01 16:01:46.991900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:46.991845 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.050276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.050229 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.087775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.084084 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5"] Nov 01 16:01:47.087775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.084267 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.139780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.136629 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.139780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.138086 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:01:47.139780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.138222 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:01:47.209524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.180229 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.209524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.180292 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9875\" (UniqueName: \"kubernetes.io/projected/6247475a-850f-47e2-8e6b-31f11e90e201-kube-api-access-p9875\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.209524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.180344 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.353845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.345164 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.353845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.345408 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9875\" (UniqueName: \"kubernetes.io/projected/6247475a-850f-47e2-8e6b-31f11e90e201-kube-api-access-p9875\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.353845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.345475 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.468068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.467971 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.471775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.468299 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:47.497954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.497899 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.527777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.440738 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.538193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.538140 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.541772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.538461 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:47.564076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.541858 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.564331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.564302 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:47.564660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.559479 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.564832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.564811 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:47.682420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.681068 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9875\" (UniqueName: \"kubernetes.io/projected/6247475a-850f-47e2-8e6b-31f11e90e201-kube-api-access-p9875\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.682420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.681196 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.766836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.686792 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:01:47.874050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.690584 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bbf71bf-skbj5\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.874272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.744126 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:01:47.874423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.744391 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:01:47.874550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.767122 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:47.874667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.824903 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:47.875114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.875094 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:47.877821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:47.874786 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:48.078429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.070936 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.078429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.072037 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:01:48.139484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.138906 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.156020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.139857 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:48.227778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.224307 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.227778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.224364 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:48.420776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.372047 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.523919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.523874 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:48.992858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.543371 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.992858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.554101 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:48.992858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.989654 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:48.992858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.701856 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:48.992858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.683842 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:01:49.044363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.588439 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.195933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.195893 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:49.205856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.077851 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.206384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.061920 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.206553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.077917 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.206948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:48.588487 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.207149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.207117 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:49.207278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.194066 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:49.207452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.077892 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:01:49.207813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.077936 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:01:49.222784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.209600 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Nov 01 16:01:49.224017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.223992 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.224192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.224170 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:49.225545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.225516 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:49.225642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.225580 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:01:49.225642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.225618 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:01:49.225781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.225657 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:01:49.225781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.225690 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:01:49.257965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.257870 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:49.380317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.380279 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.380592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.380564 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:49.471962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.447820 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.649894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.649847 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:49.650149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.474811 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:01:49.651246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.495801 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.677914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.677868 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:49.678186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.582207 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:49.678318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:49.678295 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:50.012129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.006922 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:01:50.012129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.011816 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:01:50.012277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.012186 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:01:50.012944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.012921 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:01:50.152089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.060882 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:50.152089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.060986 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:50.152089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.121422 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:50.244774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.238860 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:01:50.309683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.254488 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:01:50.431451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.254522 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:01:50.431451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.254546 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:01:50.431451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.257256 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.431451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.403481 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:50.431451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.381729 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.431451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.403554 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:50.431451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.312930 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:01:50.580013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.579921 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.683964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.683892 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:50.683964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.580258 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.684110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.683987 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:50.684110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.637822 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:50.684110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.684034 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:50.767846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:50.767643 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:51.022935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.022891 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.024381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.023115 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.029810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.023144 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:01:51.081819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.067434 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.083409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.083381 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:01:51.298654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.293875 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:01:51.337997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.337959 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.338269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.338228 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:51.391782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.386857 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5"] Nov 01 16:01:51.416775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.403844 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.416775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.406161 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.416775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.411917 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.416775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.415348 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.416775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.415393 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:51.416775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.415939 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:51.453886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.453846 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.454162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.454132 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:51.497981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.497945 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.498238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.498212 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:51.690775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.689863 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:51.690775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.689924 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:51.997840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:51.994539 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" event={"ID":"6247475a-850f-47e2-8e6b-31f11e90e201","Type":"ContainerStarted","Data":"0a0728b0bc50670f56c4440f53de0c9ab716f164d7fa40032a6be1f8bdc90015"} Nov 01 16:01:52.039775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.030108 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.041322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.041267 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.048778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.043131 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:01:52.093951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.087360 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.224846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.224193 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.224846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.224256 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:52.295908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.295860 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.371679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.371636 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.372296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.372247 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:52.392878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.392833 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 01 16:01:52.408870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.408826 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.409485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.409111 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.414836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.414805 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:01:52.419193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.419167 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:52.448221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.448182 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.448515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.448483 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:52.471288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.471252 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.471583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.471538 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:52.654777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.631587 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:52.659169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.659132 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:52.839149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:52.839098 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.044136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.044092 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.050779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.044394 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:01:53.065565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.065526 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.089345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.089297 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.101627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.101592 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" event={"ID":"6247475a-850f-47e2-8e6b-31f11e90e201","Type":"ContainerStarted","Data":"ba80226fde011e6a4365fef0196ab2c062be783d4d4e4b4c53a63b705e5cf09e"} Nov 01 16:01:53.227241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.227144 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.227476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.227448 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:53.310906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.310855 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.374552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.374108 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.374552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.374167 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:53.418775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.417136 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.418775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.417526 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:01:53.418775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.417821 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:01:53.423775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.422831 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:53.451834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.451799 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.452139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.452093 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:53.466187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.466145 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.466396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.466375 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:53.484997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.484906 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 01 16:01:53.587795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.580234 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:53.587795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:53.580296 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:54.050948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.050890 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.051632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.051598 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.070095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.070055 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.097272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.097228 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.246787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.216436 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.247118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.247076 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:54.347318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.347219 4573 generic.go:356] "Generic (PLEG): container finished" podID="6247475a-850f-47e2-8e6b-31f11e90e201" containerID="ba80226fde011e6a4365fef0196ab2c062be783d4d4e4b4c53a63b705e5cf09e" exitCode=0 Nov 01 16:01:54.347822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.347791 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" event={"ID":"6247475a-850f-47e2-8e6b-31f11e90e201","Type":"ContainerDied","Data":"ba80226fde011e6a4365fef0196ab2c062be783d4d4e4b4c53a63b705e5cf09e"} Nov 01 16:01:54.373775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.348132 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.389170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.389135 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.411880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.411827 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:54.420505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.420465 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:01:54.425430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.420717 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.646219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.425049 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.647091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.451177 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.647275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.647249 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:54.647377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.624572 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Nov 01 16:01:54.647482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.461861 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:54.647823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.566411 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.647982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.647958 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:54.710955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.710922 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:54.711214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.711188 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:54.975098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:54.974984 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" podUID="09f03273-d479-473f-8a43-c1345ce65fe7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.24:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.022106 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:01:55.022064 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:01:55.097774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.080928 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:01:55.097774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.081426 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:01:55.097774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.081773 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.098918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.098859 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.232960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.224134 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.232960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.224194 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:55.351329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.351281 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.389518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.389471 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.390154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.390110 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:55.426978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.425838 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.535172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.528806 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.535172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.528886 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:55.698508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.566231 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.700135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.700080 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:55.725544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.667705 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.726122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.667899 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:01:55.727116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.667667 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:55.917777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.664077 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:55.917777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:55.916888 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:56.046942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.023885 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 01 16:01:56.101206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.089886 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:01:56.133762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.107590 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.133762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.114308 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:01:56.150772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.139817 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:01:56.232821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.229385 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:56.352777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.243689 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.352777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.243774 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:56.353185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.353143 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.439728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.439679 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:01:56.479891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.479851 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.480177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.480146 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:56.525830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.504204 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.525830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.504290 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:56.571620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.571578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:56.692776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.661921 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-openshift-ca\") pod \"6247475a-850f-47e2-8e6b-31f11e90e201\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " Nov 01 16:01:56.727215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.727144 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:01:56.727925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.727868 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:56.763059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.710505 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-openshift-ca" pod "6247475a-850f-47e2-8e6b-31f11e90e201" (UID: "6247475a-850f-47e2-8e6b-31f11e90e201"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:01:56.763059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.710654 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:56.763059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.747764 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:56.766198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.715500 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-xks-ca\") pod \"6247475a-850f-47e2-8e6b-31f11e90e201\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " Nov 01 16:01:56.766624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.766579 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6247475a-850f-47e2-8e6b-31f11e90e201-kube-api-access-p9875\" (UniqueName: \"kubernetes.io/projected/6247475a-850f-47e2-8e6b-31f11e90e201-kube-api-access-p9875\") pod \"6247475a-850f-47e2-8e6b-31f11e90e201\" (UID: \"6247475a-850f-47e2-8e6b-31f11e90e201\") " Nov 01 16:01:56.770849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:56.770826 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:57.048173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.048110 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Nov 01 16:01:57.102852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.102805 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.105194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.105162 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6247475a-850f-47e2-8e6b-31f11e90e201-kube-api-access-p9875" pod "6247475a-850f-47e2-8e6b-31f11e90e201" (UID: "6247475a-850f-47e2-8e6b-31f11e90e201"). InnerVolumeSpecName "kube-api-access-p9875". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:01:57.203777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.176016 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-xks-ca" pod "6247475a-850f-47e2-8e6b-31f11e90e201" (UID: "6247475a-850f-47e2-8e6b-31f11e90e201"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:01:57.279933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.279826 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:01:57.301909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.301870 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.307869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.307819 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:57.308443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.308420 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.308595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.308576 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:57.308718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.308700 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.309684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.309656 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:01:57.409810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.313182 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9875\" (UniqueName: \"kubernetes.io/projected/6247475a-850f-47e2-8e6b-31f11e90e201-kube-api-access-p9875\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:57.409810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.322474 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.409810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.356825 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.409810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.378779 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" Nov 01 16:01:57.409810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.382264 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.409810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.400628 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:57.643843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.566952 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:01:57.746323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.669387 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6247475a-850f-47e2-8e6b-31f11e90e201-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:01:57.746546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.746509 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.746844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.746816 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.762357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.761939 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6dlmn" podUID="09f03273-d479-473f-8a43-c1345ce65fe7" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.24:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:57.906009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.883144 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.906009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.883204 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:57.906009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.883293 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.906009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.883322 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:57.906009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.883434 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:57.906009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:57.883460 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:58.043025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.041228 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:58.103160 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:01:58.103128 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.499s" Nov 01 16:01:58.107952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.107927 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5" event={"ID":"6247475a-850f-47e2-8e6b-31f11e90e201","Type":"ContainerDied","Data":"0a0728b0bc50670f56c4440f53de0c9ab716f164d7fa40032a6be1f8bdc90015"} Nov 01 16:01:58.108152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.108135 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a0728b0bc50670f56c4440f53de0c9ab716f164d7fa40032a6be1f8bdc90015" Nov 01 16:01:58.108766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.103673 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:01:58.209450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.209357 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:01:58.221273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.220848 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.221273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.220914 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Nov 01 16:01:58.380272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.380247 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:01:58.384060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.384031 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.384467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.384421 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:58.408055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.384764 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:58.468017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.467915 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:01:58.470334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.470305 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:01:58.500999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.500953 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.501293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.501245 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Nov 01 16:01:58.508246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.508207 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.508439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.508415 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Nov 01 16:01:58.575433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.575387 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:58.603564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.603531 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" start-of-body= Nov 01 16:01:58.603844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.603799 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": dial tcp 10.133.0.92:15021: connect: connection refused" Nov 01 16:01:58.792589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.792497 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:01:58.933539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.933513 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:01:58.935828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.935808 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:01:58.938265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.938239 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:01:58.940046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.940030 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:01:58.940893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.940858 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:01:58.941615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.941599 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:01:58.942951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:58.942936 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:01:59.306389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.302950 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:01:59.380475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.380434 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:01:59.382388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.382356 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Nov 01 16:01:59.382629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.382602 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Nov 01 16:01:59.463628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.463581 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:01:59.465666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.465645 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:59.612832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.563351 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:01:59.647412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.601492 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:01:59.908818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.907813 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:01:59.911053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:01:59.911022 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:02:00.259196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.259062 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:02:00.287353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.287043 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:02:00.328775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.326155 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:00.328775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.326214 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:00.570828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.570167 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:02:00.579803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.573152 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:02:00.597651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.597619 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf"] Nov 01 16:02:00.599635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.599608 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6247475a-850f-47e2-8e6b-31f11e90e201" containerName="revoke-keys" Nov 01 16:02:00.599808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.599792 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6247475a-850f-47e2-8e6b-31f11e90e201" containerName="revoke-keys" Nov 01 16:02:00.600152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.600123 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6247475a-850f-47e2-8e6b-31f11e90e201" containerName="revoke-keys" Nov 01 16:02:00.603865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.603839 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:00.604026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.603998 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:00.623809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.604727 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.625375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.625330 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:02:00.632915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.632889 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:02:00.633074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.632946 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf"] Nov 01 16:02:00.781900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.781835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:00.786605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.786578 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.786854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.786832 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmqgh\" (UniqueName: \"kubernetes.io/projected/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-kube-api-access-bmqgh\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.787061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.787043 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.901774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.896720 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.901774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.896798 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmqgh\" (UniqueName: \"kubernetes.io/projected/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-kube-api-access-bmqgh\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.901774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.896903 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.901774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.899592 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.956439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.956403 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:00.989835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:00.989798 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmqgh\" (UniqueName: \"kubernetes.io/projected/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-kube-api-access-bmqgh\") pod \"maas-api-revoke-sub-e50165da-2x4xf\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:01.270117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:01.270016 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:01.380426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:01.379911 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:01.380692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:01.380652 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:01.478256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:01.478221 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:01.673900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:01.673852 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:02:01.674211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:01.674171 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:01.681150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:01.681121 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:02:01.715250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:01.715220 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:02:02.052694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:02.052664 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf"] Nov 01 16:02:02.053137 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:02.052902 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af195cd_dfeb_4ba2_b4d2_f7f832c94938.slice/crio-86f4094abb99ed967c9b16d5a5dd9f80c5cf442c42ef505ec2411786a14dd09e WatchSource:0}: Error finding container 86f4094abb99ed967c9b16d5a5dd9f80c5cf442c42ef505ec2411786a14dd09e: Status 404 returned error can't find the container with id 86f4094abb99ed967c9b16d5a5dd9f80c5cf442c42ef505ec2411786a14dd09e Nov 01 16:02:02.165488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:02.165447 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:02:02.168203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:02.168175 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:02:02.277899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:02.277862 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:02.827363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:02.827275 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" event={"ID":"1af195cd-dfeb-4ba2-b4d2-f7f832c94938","Type":"ContainerStarted","Data":"6029b718d6f9535d488b974189b001de8987b27eaed3a42bccaa9200199641d9"} Nov 01 16:02:02.827363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:02.827324 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" event={"ID":"1af195cd-dfeb-4ba2-b4d2-f7f832c94938","Type":"ContainerStarted","Data":"86f4094abb99ed967c9b16d5a5dd9f80c5cf442c42ef505ec2411786a14dd09e"} Nov 01 16:02:02.844469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:02.844422 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" podStartSLOduration=2.8444080659999997 podStartE2EDuration="2.844408066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:02.841280617 +0000 UTC m=+860.544446552" watchObservedRunningTime="2026-11-01 16:02:02.844408066 +0000 UTC m=+860.547574002" Nov 01 16:02:03.074570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:03.074532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:03.761659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:03.761625 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:03.842085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:03.842047 4573 generic.go:356] "Generic (PLEG): container finished" podID="1af195cd-dfeb-4ba2-b4d2-f7f832c94938" containerID="6029b718d6f9535d488b974189b001de8987b27eaed3a42bccaa9200199641d9" exitCode=0 Nov 01 16:02:03.842269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:03.842123 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" event={"ID":"1af195cd-dfeb-4ba2-b4d2-f7f832c94938","Type":"ContainerDied","Data":"6029b718d6f9535d488b974189b001de8987b27eaed3a42bccaa9200199641d9"} Nov 01 16:02:04.571762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:04.571730 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:05.010173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.010143 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:05.084197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.084158 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-kube-api-access-bmqgh\" (UniqueName: \"kubernetes.io/projected/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-kube-api-access-bmqgh\") pod \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " Nov 01 16:02:05.084418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.084400 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-xks-ca\") pod \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " Nov 01 16:02:05.084490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.084462 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-openshift-ca\") pod \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\" (UID: \"1af195cd-dfeb-4ba2-b4d2-f7f832c94938\") " Nov 01 16:02:05.084889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.084858 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-openshift-ca" pod "1af195cd-dfeb-4ba2-b4d2-f7f832c94938" (UID: "1af195cd-dfeb-4ba2-b4d2-f7f832c94938"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:05.085176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.085155 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:05.086392 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.086369 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-kube-api-access-bmqgh" pod "1af195cd-dfeb-4ba2-b4d2-f7f832c94938" (UID: "1af195cd-dfeb-4ba2-b4d2-f7f832c94938"). InnerVolumeSpecName "kube-api-access-bmqgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:05.086519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.086494 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-xks-ca" pod "1af195cd-dfeb-4ba2-b4d2-f7f832c94938" (UID: "1af195cd-dfeb-4ba2-b4d2-f7f832c94938"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:05.187021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.186990 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmqgh\" (UniqueName: \"kubernetes.io/projected/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-kube-api-access-bmqgh\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:05.187021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.187019 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af195cd-dfeb-4ba2-b4d2-f7f832c94938-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:05.256728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.256680 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:05.862432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.862404 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" Nov 01 16:02:05.862854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.862405 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf" event={"ID":"1af195cd-dfeb-4ba2-b4d2-f7f832c94938","Type":"ContainerDied","Data":"86f4094abb99ed967c9b16d5a5dd9f80c5cf442c42ef505ec2411786a14dd09e"} Nov 01 16:02:05.862854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:05.862522 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86f4094abb99ed967c9b16d5a5dd9f80c5cf442c42ef505ec2411786a14dd09e" Nov 01 16:02:06.096955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:06.096917 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:06.778034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:06.778006 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:07.485558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:07.485524 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:08.375552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.375509 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:08.470128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.470095 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8"] Nov 01 16:02:08.471627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.471599 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1af195cd-dfeb-4ba2-b4d2-f7f832c94938" containerName="revoke-keys" Nov 01 16:02:08.471627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.471625 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af195cd-dfeb-4ba2-b4d2-f7f832c94938" containerName="revoke-keys" Nov 01 16:02:08.471931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.471835 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af195cd-dfeb-4ba2-b4d2-f7f832c94938" containerName="revoke-keys" Nov 01 16:02:08.478982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.478961 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.481113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.481087 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8"] Nov 01 16:02:08.482863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.482838 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:02:08.483209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.483174 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:02:08.546137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.546102 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.546536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.546209 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngrfq\" (UniqueName: \"kubernetes.io/projected/cc086bef-b8c6-4566-8e03-0d40c08ec484-kube-api-access-ngrfq\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.546536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.546389 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.648215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.648137 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.648381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.648220 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngrfq\" (UniqueName: \"kubernetes.io/projected/cc086bef-b8c6-4566-8e03-0d40c08ec484-kube-api-access-ngrfq\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.648381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.648287 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.648944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.648918 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.650510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.650488 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.656495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.656472 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngrfq\" (UniqueName: \"kubernetes.io/projected/cc086bef-b8c6-4566-8e03-0d40c08ec484-kube-api-access-ngrfq\") pod \"maas-api-revoke-sub-8d182e87-l8vs8\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:08.799250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:08.799219 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:09.079165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:09.079130 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8"] Nov 01 16:02:09.080138 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:09.080111 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc086bef_b8c6_4566_8e03_0d40c08ec484.slice/crio-033584e1da634244ec366b185c20bb4ab7e5e758b4cc2059fb29f76fb5593e2d WatchSource:0}: Error finding container 033584e1da634244ec366b185c20bb4ab7e5e758b4cc2059fb29f76fb5593e2d: Status 404 returned error can't find the container with id 033584e1da634244ec366b185c20bb4ab7e5e758b4cc2059fb29f76fb5593e2d Nov 01 16:02:09.275946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:09.275915 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:09.376072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:09.376040 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:09.484908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:09.484863 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:09.904847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:09.904807 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" event={"ID":"cc086bef-b8c6-4566-8e03-0d40c08ec484","Type":"ContainerStarted","Data":"37bce1c9eafd62f7b38d097d3043116f60d3a9b1fcd810dd1c0810089646bec3"} Nov 01 16:02:09.905307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:09.904901 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" event={"ID":"cc086bef-b8c6-4566-8e03-0d40c08ec484","Type":"ContainerStarted","Data":"033584e1da634244ec366b185c20bb4ab7e5e758b4cc2059fb29f76fb5593e2d"} Nov 01 16:02:09.923786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:09.923715 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" podStartSLOduration=1.923701895 podStartE2EDuration="1.923701895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:09.919097183 +0000 UTC m=+867.622263118" watchObservedRunningTime="2026-11-01 16:02:09.923701895 +0000 UTC m=+867.626867830" Nov 01 16:02:10.118253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:10.118220 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:10.913163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:10.913130 4573 generic.go:356] "Generic (PLEG): container finished" podID="cc086bef-b8c6-4566-8e03-0d40c08ec484" containerID="37bce1c9eafd62f7b38d097d3043116f60d3a9b1fcd810dd1c0810089646bec3" exitCode=0 Nov 01 16:02:10.913635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:10.913246 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" event={"ID":"cc086bef-b8c6-4566-8e03-0d40c08ec484","Type":"ContainerDied","Data":"37bce1c9eafd62f7b38d097d3043116f60d3a9b1fcd810dd1c0810089646bec3"} Nov 01 16:02:11.057933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:11.057872 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:12.082441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.082413 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:12.207431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.205608 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:12.215775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.212590 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-openshift-ca\") pod \"cc086bef-b8c6-4566-8e03-0d40c08ec484\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " Nov 01 16:02:12.215775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.212900 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc086bef-b8c6-4566-8e03-0d40c08ec484-kube-api-access-ngrfq\" (UniqueName: \"kubernetes.io/projected/cc086bef-b8c6-4566-8e03-0d40c08ec484-kube-api-access-ngrfq\") pod \"cc086bef-b8c6-4566-8e03-0d40c08ec484\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " Nov 01 16:02:12.215775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.213125 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-xks-ca\") pod \"cc086bef-b8c6-4566-8e03-0d40c08ec484\" (UID: \"cc086bef-b8c6-4566-8e03-0d40c08ec484\") " Nov 01 16:02:12.222775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.218290 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-openshift-ca" pod "cc086bef-b8c6-4566-8e03-0d40c08ec484" (UID: "cc086bef-b8c6-4566-8e03-0d40c08ec484"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:12.222775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.218569 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:12.244118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.243868 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-xks-ca" pod "cc086bef-b8c6-4566-8e03-0d40c08ec484" (UID: "cc086bef-b8c6-4566-8e03-0d40c08ec484"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:12.245682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.245635 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc086bef-b8c6-4566-8e03-0d40c08ec484-kube-api-access-ngrfq" pod "cc086bef-b8c6-4566-8e03-0d40c08ec484" (UID: "cc086bef-b8c6-4566-8e03-0d40c08ec484"). InnerVolumeSpecName "kube-api-access-ngrfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:12.320214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.320178 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc086bef-b8c6-4566-8e03-0d40c08ec484-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:12.320214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.320210 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngrfq\" (UniqueName: \"kubernetes.io/projected/cc086bef-b8c6-4566-8e03-0d40c08ec484-kube-api-access-ngrfq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:12.930480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.930441 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" event={"ID":"cc086bef-b8c6-4566-8e03-0d40c08ec484","Type":"ContainerDied","Data":"033584e1da634244ec366b185c20bb4ab7e5e758b4cc2059fb29f76fb5593e2d"} Nov 01 16:02:12.930632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.930494 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033584e1da634244ec366b185c20bb4ab7e5e758b4cc2059fb29f76fb5593e2d" Nov 01 16:02:12.930632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:12.930454 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8" Nov 01 16:02:13.017326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.017290 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:13.498391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.498358 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8"] Nov 01 16:02:13.499713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.499685 4573 cpu_manager.go:414] "containerMap: removing container" podUID="cc086bef-b8c6-4566-8e03-0d40c08ec484" containerName="revoke-keys" Nov 01 16:02:13.499713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.499709 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc086bef-b8c6-4566-8e03-0d40c08ec484" containerName="revoke-keys" Nov 01 16:02:13.499959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.499873 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc086bef-b8c6-4566-8e03-0d40c08ec484" containerName="revoke-keys" Nov 01 16:02:13.505088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.504995 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.506510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.506482 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8"] Nov 01 16:02:13.638158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638110 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.638342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638169 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.638342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638214 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.638342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638262 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxt8d\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-kube-api-access-qxt8d\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.638482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638464 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.638607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638586 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.638641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638626 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.638677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638652 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.638830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.638814 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e19be-9c70-4a83-8880-d4d5b4251407-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.650442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.650412 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m"] Nov 01 16:02:13.654851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.654832 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.668981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.668960 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m"] Nov 01 16:02:13.740489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.740456 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e849359-8086-44e6-8a61-b879af942f35-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.740642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.740521 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.740642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.740621 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.740718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.740659 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.740718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.740680 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.740835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.740721 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.740907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.740884 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.740992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.740971 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.741055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741005 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-data\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741043 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.741132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741055 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741109 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.741190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741155 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e19be-9c70-4a83-8880-d4d5b4251407-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741223 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741255 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741284 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2gr\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-kube-api-access-rd2gr\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.741317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741310 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741352 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxt8d\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-kube-api-access-qxt8d\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741467 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741504 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e849359-8086-44e6-8a61-b879af942f35-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.741617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741590 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-certs\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741716 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-credential-socket\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.741853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.741831 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e19be-9c70-4a83-8880-d4d5b4251407-istiod-ca-cert\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.743608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.743587 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-envoy\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.743836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.743820 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-podinfo\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.749881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.749833 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-token\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.750097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.750076 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxt8d\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-kube-api-access-qxt8d\") pod \"e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.786606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.786578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:13.825673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.825646 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:13.860801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.860608 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.860801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.860738 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.860816 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.860854 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.860897 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.861027 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2gr\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-kube-api-access-rd2gr\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.861140 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-certs\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.861205 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e849359-8086-44e6-8a61-b879af942f35-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.861625 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e849359-8086-44e6-8a61-b879af942f35-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.861736 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.861919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.861901 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.862142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.862118 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-credential-socket\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.862375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.862345 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-data\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.862469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.862439 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e849359-8086-44e6-8a61-b879af942f35-istiod-ca-cert\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.864190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.864153 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-envoy\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.864439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.864420 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e849359-8086-44e6-8a61-b879af942f35-istio-podinfo\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.870625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.870597 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-istio-token\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.871980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.871957 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2gr\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-kube-api-access-rd2gr\") pod \"e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:13.973733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:13.973699 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:14.144076 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:14.142723 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3e19be_9c70_4a83_8880_d4d5b4251407.slice/crio-54c1a8e62fc421d3c2ee4b17e7d9111038710eb770461db1da70814ba75c42cd WatchSource:0}: Error finding container 54c1a8e62fc421d3c2ee4b17e7d9111038710eb770461db1da70814ba75c42cd: Status 404 returned error can't find the container with id 54c1a8e62fc421d3c2ee4b17e7d9111038710eb770461db1da70814ba75c42cd Nov 01 16:02:14.144979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.144953 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8"] Nov 01 16:02:14.146008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.145966 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.146102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.146033 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.146102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.146062 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.308885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.308857 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m"] Nov 01 16:02:14.309015 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:14.308995 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e849359_8086_44e6_8a61_b879af942f35.slice/crio-9aac7740a4257614fd4e135967c41d920dfd4825b06273903fe33b4e4e5f161f WatchSource:0}: Error finding container 9aac7740a4257614fd4e135967c41d920dfd4825b06273903fe33b4e4e5f161f: Status 404 returned error can't find the container with id 9aac7740a4257614fd4e135967c41d920dfd4825b06273903fe33b4e4e5f161f Nov 01 16:02:14.311732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.311700 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.311900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.311784 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.311900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.311836 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:14.765400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.760239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:14.951683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.951649 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" event={"ID":"9e849359-8086-44e6-8a61-b879af942f35","Type":"ContainerStarted","Data":"9a7153d1128a488a008af9e546ac49cdfeaf0592f96d0b61f701629e07c4784e"} Nov 01 16:02:14.951879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.951701 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" event={"ID":"9e849359-8086-44e6-8a61-b879af942f35","Type":"ContainerStarted","Data":"9aac7740a4257614fd4e135967c41d920dfd4825b06273903fe33b4e4e5f161f"} Nov 01 16:02:14.953390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.953361 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" event={"ID":"0d3e19be-9c70-4a83-8880-d4d5b4251407","Type":"ContainerStarted","Data":"43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1"} Nov 01 16:02:14.953509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.953415 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" event={"ID":"0d3e19be-9c70-4a83-8880-d4d5b4251407","Type":"ContainerStarted","Data":"54c1a8e62fc421d3c2ee4b17e7d9111038710eb770461db1da70814ba75c42cd"} Nov 01 16:02:14.980783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:14.974516 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:15.011974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:15.011905 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podStartSLOduration=2.011879929 podStartE2EDuration="2.011879929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:14.988023132 +0000 UTC m=+872.691189081" watchObservedRunningTime="2026-11-01 16:02:15.011879929 +0000 UTC m=+872.715045866" Nov 01 16:02:15.031889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:15.031786 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podStartSLOduration=2.031771333 podStartE2EDuration="2.031771333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:15.030382765 +0000 UTC m=+872.733548704" watchObservedRunningTime="2026-11-01 16:02:15.031771333 +0000 UTC m=+872.734937269" Nov 01 16:02:15.826276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:15.826243 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:15.827761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:15.827728 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:02:15.827852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:15.827799 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 01 16:02:15.932078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:15.932048 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:15.986774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:15.985843 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:02:15.986774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:15.985901 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 16:02:16.826249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:16.826209 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.826414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:16.826271 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 01 16:02:16.977306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:16.977269 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:02:16.977580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:16.977548 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 16:02:17.228134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:17.228088 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:17.826995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:17.826957 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:02:17.827627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:17.827009 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 01 16:02:18.015775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:18.015001 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.015775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:18.015086 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 16:02:18.826804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:18.826767 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.826956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:18.826829 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 01 16:02:18.975775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:18.971259 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:18.992786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:18.983125 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:02:18.992786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:18.983183 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 16:02:19.831321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:19.831159 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.831321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:19.831242 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 01 16:02:19.975618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:19.974196 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:02:19.975618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:19.974260 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 16:02:20.826874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:20.826826 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.827074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:20.826889 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Nov 01 16:02:20.974210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:20.974174 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Nov 01 16:02:20.974386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:20.974230 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Nov 01 16:02:21.576903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:21.576358 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:21.831051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:21.830972 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:21.831483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:21.831464 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:21.832252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:21.832232 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:02:22.044650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:22.044615 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:22.045653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:22.045618 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:22.046473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:22.046442 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:02:23.284401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:23.284349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:24.193065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:24.193031 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:24.366148 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:02:24.366113 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:02:25.065426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:25.065392 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:25.778555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:25.778521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:26.586786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:26.586733 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:27.378276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.378247 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:27.710866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.710780 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx"] Nov 01 16:02:27.715104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.715087 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.718475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.718453 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:02:27.719282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.719261 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:02:27.722607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.722581 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx"] Nov 01 16:02:27.784219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.784175 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2vl\" (UniqueName: \"kubernetes.io/projected/81312c68-0f0e-46be-8c7f-64147d6e23eb-kube-api-access-vs2vl\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.784391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.784380 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.784437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.784407 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.886085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.886054 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2vl\" (UniqueName: \"kubernetes.io/projected/81312c68-0f0e-46be-8c7f-64147d6e23eb-kube-api-access-vs2vl\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.886249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.886223 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.886291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.886251 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.886983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.886960 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.888526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.888506 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:27.893422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:27.893395 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2vl\" (UniqueName: \"kubernetes.io/projected/81312c68-0f0e-46be-8c7f-64147d6e23eb-kube-api-access-vs2vl\") pod \"maas-api-revoke-sub-aa6f3092-sgkcx\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:28.035657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:28.035578 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:28.272627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:28.272581 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:28.324968 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:28.324928 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81312c68_0f0e_46be_8c7f_64147d6e23eb.slice/crio-af1fd0d3e57bd9df5657d8b6c1af8320e396f631ebf6934353416ec0b02a3c95 WatchSource:0}: Error finding container af1fd0d3e57bd9df5657d8b6c1af8320e396f631ebf6934353416ec0b02a3c95: Status 404 returned error can't find the container with id af1fd0d3e57bd9df5657d8b6c1af8320e396f631ebf6934353416ec0b02a3c95 Nov 01 16:02:28.325920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:28.325890 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx"] Nov 01 16:02:29.091542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:29.091502 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:29.141507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:29.141469 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" event={"ID":"81312c68-0f0e-46be-8c7f-64147d6e23eb","Type":"ContainerStarted","Data":"8a2453b2e4a805cf8a8960e29faf9d1484b93f43d5683567a0debd1406b8d80e"} Nov 01 16:02:29.141678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:29.141538 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" event={"ID":"81312c68-0f0e-46be-8c7f-64147d6e23eb","Type":"ContainerStarted","Data":"af1fd0d3e57bd9df5657d8b6c1af8320e396f631ebf6934353416ec0b02a3c95"} Nov 01 16:02:29.271820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:29.263561 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" podStartSLOduration=2.263541336 podStartE2EDuration="2.263541336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:29.172912778 +0000 UTC m=+886.876078754" watchObservedRunningTime="2026-11-01 16:02:29.263541336 +0000 UTC m=+886.966707273" Nov 01 16:02:30.547020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:30.546965 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:30.549648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:30.549625 4573 generic.go:356] "Generic (PLEG): container finished" podID="81312c68-0f0e-46be-8c7f-64147d6e23eb" containerID="8a2453b2e4a805cf8a8960e29faf9d1484b93f43d5683567a0debd1406b8d80e" exitCode=0 Nov 01 16:02:30.549930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:30.549906 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" event={"ID":"81312c68-0f0e-46be-8c7f-64147d6e23eb","Type":"ContainerDied","Data":"8a2453b2e4a805cf8a8960e29faf9d1484b93f43d5683567a0debd1406b8d80e"} Nov 01 16:02:30.715344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:30.715287 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:31.264775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:31.259833 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:02:31.293047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:31.291824 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:02:31.421778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:31.418934 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:31.421778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:31.419285 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:02:32.068335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.068292 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.264355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.264315 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:02:32.276129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.265228 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:02:32.397214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.276846 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:02:32.397682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.394099 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.471116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.470830 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:02:32.564905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.554183 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:02:32.567196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.565717 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:02:32.627143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.626134 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:02:32.651432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.651405 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:32.748399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.748372 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-openshift-ca\") pod \"81312c68-0f0e-46be-8c7f-64147d6e23eb\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " Nov 01 16:02:32.748574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.748468 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-xks-ca\") pod \"81312c68-0f0e-46be-8c7f-64147d6e23eb\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " Nov 01 16:02:32.748574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.748553 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81312c68-0f0e-46be-8c7f-64147d6e23eb-kube-api-access-vs2vl\" (UniqueName: \"kubernetes.io/projected/81312c68-0f0e-46be-8c7f-64147d6e23eb-kube-api-access-vs2vl\") pod \"81312c68-0f0e-46be-8c7f-64147d6e23eb\" (UID: \"81312c68-0f0e-46be-8c7f-64147d6e23eb\") " Nov 01 16:02:32.748739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.748713 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-openshift-ca" pod "81312c68-0f0e-46be-8c7f-64147d6e23eb" (UID: "81312c68-0f0e-46be-8c7f-64147d6e23eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:32.749286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.749264 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:32.750667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.750631 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-xks-ca" pod "81312c68-0f0e-46be-8c7f-64147d6e23eb" (UID: "81312c68-0f0e-46be-8c7f-64147d6e23eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:32.750746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.750684 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81312c68-0f0e-46be-8c7f-64147d6e23eb-kube-api-access-vs2vl" pod "81312c68-0f0e-46be-8c7f-64147d6e23eb" (UID: "81312c68-0f0e-46be-8c7f-64147d6e23eb"). InnerVolumeSpecName "kube-api-access-vs2vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:32.828216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.828132 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" Nov 01 16:02:32.828216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.828166 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx" event={"ID":"81312c68-0f0e-46be-8c7f-64147d6e23eb","Type":"ContainerDied","Data":"af1fd0d3e57bd9df5657d8b6c1af8320e396f631ebf6934353416ec0b02a3c95"} Nov 01 16:02:32.828216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.828216 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af1fd0d3e57bd9df5657d8b6c1af8320e396f631ebf6934353416ec0b02a3c95" Nov 01 16:02:32.850560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.850507 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81312c68-0f0e-46be-8c7f-64147d6e23eb-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:32.850560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:32.850536 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs2vl\" (UniqueName: \"kubernetes.io/projected/81312c68-0f0e-46be-8c7f-64147d6e23eb-kube-api-access-vs2vl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:33.295923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:33.295887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:33.993065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:33.993035 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g"] Nov 01 16:02:33.993681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:33.993667 4573 cpu_manager.go:414] "containerMap: removing container" podUID="81312c68-0f0e-46be-8c7f-64147d6e23eb" containerName="revoke-keys" Nov 01 16:02:33.993730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:33.993682 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81312c68-0f0e-46be-8c7f-64147d6e23eb" containerName="revoke-keys" Nov 01 16:02:33.993858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:33.993845 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="81312c68-0f0e-46be-8c7f-64147d6e23eb" containerName="revoke-keys" Nov 01 16:02:33.997558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:33.997540 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.001425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.000635 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:02:34.001528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.001435 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:02:34.004994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.004973 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g"] Nov 01 16:02:34.070993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.070964 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.070993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.070995 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf9dz\" (UniqueName: \"kubernetes.io/projected/7ad054e7-5660-4df2-86ff-5fc8cd765683-kube-api-access-vf9dz\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.071227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.071207 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.173174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.173128 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.173455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.173429 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf9dz\" (UniqueName: \"kubernetes.io/projected/7ad054e7-5660-4df2-86ff-5fc8cd765683-kube-api-access-vf9dz\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.173795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.173768 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.174501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.174474 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.177208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.177182 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-xks-ca\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.182198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.182169 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf9dz\" (UniqueName: \"kubernetes.io/projected/7ad054e7-5660-4df2-86ff-5fc8cd765683-kube-api-access-vf9dz\") pod \"maas-api-revoke-sub-26c559d0-b2m4g\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.318039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.317958 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:34.486291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.486253 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2"] Nov 01 16:02:34.489488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.489460 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:34.492957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.492936 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.497073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.497032 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2"] Nov 01 16:02:34.579658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.579590 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bhrh\" (UniqueName: \"kubernetes.io/projected/9d74c586-6de5-4d16-8469-66ebaeb3b906-kube-api-access-7bhrh\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.579658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.579630 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.579857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.579661 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.606340 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:34.606310 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad054e7_5660_4df2_86ff_5fc8cd765683.slice/crio-13dd3852ebc0ffa88ca845962b09565d391a8b86271b2b2c02ebb249ac0c12d2 WatchSource:0}: Error finding container 13dd3852ebc0ffa88ca845962b09565d391a8b86271b2b2c02ebb249ac0c12d2: Status 404 returned error can't find the container with id 13dd3852ebc0ffa88ca845962b09565d391a8b86271b2b2c02ebb249ac0c12d2 Nov 01 16:02:34.606598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.606575 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g"] Nov 01 16:02:34.680864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.680840 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bhrh\" (UniqueName: \"kubernetes.io/projected/9d74c586-6de5-4d16-8469-66ebaeb3b906-kube-api-access-7bhrh\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.680989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.680890 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.680989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.680909 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.681602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.681580 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.683063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.683042 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.689616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.689594 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bhrh\" (UniqueName: \"kubernetes.io/projected/9d74c586-6de5-4d16-8469-66ebaeb3b906-kube-api-access-7bhrh\") pod \"maas-api-revoke-sub-ca58e7ae-tptf2\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.811965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.811938 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:34.847901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.847809 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" event={"ID":"7ad054e7-5660-4df2-86ff-5fc8cd765683","Type":"ContainerStarted","Data":"13dd3852ebc0ffa88ca845962b09565d391a8b86271b2b2c02ebb249ac0c12d2"} Nov 01 16:02:34.848166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.848147 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm"] Nov 01 16:02:34.855210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.855183 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.858401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.857537 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm"] Nov 01 16:02:34.882217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.882183 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:34.883640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.883607 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.883779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.883659 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.883779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.883725 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mkzf\" (UniqueName: \"kubernetes.io/projected/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-kube-api-access-4mkzf\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.988056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.988024 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.988173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.988079 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.988173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.988130 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mkzf\" (UniqueName: \"kubernetes.io/projected/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-kube-api-access-4mkzf\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.988741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.988719 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.991216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.991193 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:34.995715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:34.995690 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mkzf\" (UniqueName: \"kubernetes.io/projected/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-kube-api-access-4mkzf\") pod \"maas-api-revoke-sub-d2ea38d6-tgsgm\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:35.103727 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:35.103700 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d74c586_6de5_4d16_8469_66ebaeb3b906.slice/crio-4ae517f7844dab7446b1d1f176d83680aa60285a42b263d80a50bd58bc17e859 WatchSource:0}: Error finding container 4ae517f7844dab7446b1d1f176d83680aa60285a42b263d80a50bd58bc17e859: Status 404 returned error can't find the container with id 4ae517f7844dab7446b1d1f176d83680aa60285a42b263d80a50bd58bc17e859 Nov 01 16:02:35.104122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.104097 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2"] Nov 01 16:02:35.176230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.176201 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:35.468463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.468433 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm"] Nov 01 16:02:35.469262 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:35.469239 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4d55bc_75a1_4f9e_a45c_69bf127fddc2.slice/crio-80e7fc7d5e6b4f8d5fd4c4d00f3000ce4a55622807100d141f988b2c143f297c WatchSource:0}: Error finding container 80e7fc7d5e6b4f8d5fd4c4d00f3000ce4a55622807100d141f988b2c143f297c: Status 404 returned error can't find the container with id 80e7fc7d5e6b4f8d5fd4c4d00f3000ce4a55622807100d141f988b2c143f297c Nov 01 16:02:35.760111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.760083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:35.867061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.867024 4573 generic.go:356] "Generic (PLEG): container finished" podID="7ad054e7-5660-4df2-86ff-5fc8cd765683" containerID="ac62b6aeeae299028831cff828400a2216cd300e94ab239d096cc7b9622d81a2" exitCode=0 Nov 01 16:02:35.867241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.867109 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" event={"ID":"7ad054e7-5660-4df2-86ff-5fc8cd765683","Type":"ContainerDied","Data":"ac62b6aeeae299028831cff828400a2216cd300e94ab239d096cc7b9622d81a2"} Nov 01 16:02:35.869195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.869167 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" event={"ID":"da4d55bc-75a1-4f9e-a45c-69bf127fddc2","Type":"ContainerStarted","Data":"80fe4b551435e2623d308fc57bf2864362d46788f85e842094fd22ae9ca49d2d"} Nov 01 16:02:35.869329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.869221 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" event={"ID":"da4d55bc-75a1-4f9e-a45c-69bf127fddc2","Type":"ContainerStarted","Data":"80e7fc7d5e6b4f8d5fd4c4d00f3000ce4a55622807100d141f988b2c143f297c"} Nov 01 16:02:35.870907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.870882 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" event={"ID":"9d74c586-6de5-4d16-8469-66ebaeb3b906","Type":"ContainerStarted","Data":"fe2bd43082c47013511a455ec4e0da388bc914112c33c723a852dca5db920a88"} Nov 01 16:02:35.870998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.870928 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" event={"ID":"9d74c586-6de5-4d16-8469-66ebaeb3b906","Type":"ContainerStarted","Data":"4ae517f7844dab7446b1d1f176d83680aa60285a42b263d80a50bd58bc17e859"} Nov 01 16:02:35.900923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.900873 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" podStartSLOduration=1.900858175 podStartE2EDuration="1.900858175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:35.899347168 +0000 UTC m=+893.602513107" watchObservedRunningTime="2026-11-01 16:02:35.900858175 +0000 UTC m=+893.604024111" Nov 01 16:02:35.926109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:35.926061 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" podStartSLOduration=1.926046032 podStartE2EDuration="1.926046032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:35.922226387 +0000 UTC m=+893.625392324" watchObservedRunningTime="2026-11-01 16:02:35.926046032 +0000 UTC m=+893.629211967" Nov 01 16:02:36.564102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:36.564073 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:36.880117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:36.880080 4573 generic.go:356] "Generic (PLEG): container finished" podID="da4d55bc-75a1-4f9e-a45c-69bf127fddc2" containerID="80fe4b551435e2623d308fc57bf2864362d46788f85e842094fd22ae9ca49d2d" exitCode=0 Nov 01 16:02:36.880301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:36.880159 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" event={"ID":"da4d55bc-75a1-4f9e-a45c-69bf127fddc2","Type":"ContainerDied","Data":"80fe4b551435e2623d308fc57bf2864362d46788f85e842094fd22ae9ca49d2d"} Nov 01 16:02:36.881698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:36.881674 4573 generic.go:356] "Generic (PLEG): container finished" podID="9d74c586-6de5-4d16-8469-66ebaeb3b906" containerID="fe2bd43082c47013511a455ec4e0da388bc914112c33c723a852dca5db920a88" exitCode=0 Nov 01 16:02:36.881856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:36.881736 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" event={"ID":"9d74c586-6de5-4d16-8469-66ebaeb3b906","Type":"ContainerDied","Data":"fe2bd43082c47013511a455ec4e0da388bc914112c33c723a852dca5db920a88"} Nov 01 16:02:37.049100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.049072 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:37.119431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.119399 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-xks-ca\") pod \"7ad054e7-5660-4df2-86ff-5fc8cd765683\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " Nov 01 16:02:37.119641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.119567 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-openshift-ca\") pod \"7ad054e7-5660-4df2-86ff-5fc8cd765683\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " Nov 01 16:02:37.119641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.119620 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad054e7-5660-4df2-86ff-5fc8cd765683-kube-api-access-vf9dz\" (UniqueName: \"kubernetes.io/projected/7ad054e7-5660-4df2-86ff-5fc8cd765683-kube-api-access-vf9dz\") pod \"7ad054e7-5660-4df2-86ff-5fc8cd765683\" (UID: \"7ad054e7-5660-4df2-86ff-5fc8cd765683\") " Nov 01 16:02:37.119942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.119913 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-openshift-ca" pod "7ad054e7-5660-4df2-86ff-5fc8cd765683" (UID: "7ad054e7-5660-4df2-86ff-5fc8cd765683"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:37.120367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.120347 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:37.121694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.121669 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-xks-ca" pod "7ad054e7-5660-4df2-86ff-5fc8cd765683" (UID: "7ad054e7-5660-4df2-86ff-5fc8cd765683"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:37.121872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.121746 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad054e7-5660-4df2-86ff-5fc8cd765683-kube-api-access-vf9dz" pod "7ad054e7-5660-4df2-86ff-5fc8cd765683" (UID: "7ad054e7-5660-4df2-86ff-5fc8cd765683"). InnerVolumeSpecName "kube-api-access-vf9dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:37.222370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.222336 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf9dz\" (UniqueName: \"kubernetes.io/projected/7ad054e7-5660-4df2-86ff-5fc8cd765683-kube-api-access-vf9dz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:37.222370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.222366 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad054e7-5660-4df2-86ff-5fc8cd765683-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:37.256890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.256856 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:37.891072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.891043 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" Nov 01 16:02:37.891500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.891087 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g" event={"ID":"7ad054e7-5660-4df2-86ff-5fc8cd765683","Type":"ContainerDied","Data":"13dd3852ebc0ffa88ca845962b09565d391a8b86271b2b2c02ebb249ac0c12d2"} Nov 01 16:02:37.891500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:37.891141 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13dd3852ebc0ffa88ca845962b09565d391a8b86271b2b2c02ebb249ac0c12d2" Nov 01 16:02:38.115628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.115602 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:38.119663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.119639 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:38.161317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.161185 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:38.243780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.243728 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-kube-api-access-4mkzf\" (UniqueName: \"kubernetes.io/projected/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-kube-api-access-4mkzf\") pod \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " Nov 01 16:02:38.243933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.243794 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-openshift-ca\") pod \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " Nov 01 16:02:38.243933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.243849 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-xks-ca\") pod \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\" (UID: \"da4d55bc-75a1-4f9e-a45c-69bf127fddc2\") " Nov 01 16:02:38.243933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.243872 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-openshift-ca\") pod \"9d74c586-6de5-4d16-8469-66ebaeb3b906\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " Nov 01 16:02:38.243933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.243896 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d74c586-6de5-4d16-8469-66ebaeb3b906-kube-api-access-7bhrh\" (UniqueName: \"kubernetes.io/projected/9d74c586-6de5-4d16-8469-66ebaeb3b906-kube-api-access-7bhrh\") pod \"9d74c586-6de5-4d16-8469-66ebaeb3b906\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " Nov 01 16:02:38.243933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.243922 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-xks-ca\") pod \"9d74c586-6de5-4d16-8469-66ebaeb3b906\" (UID: \"9d74c586-6de5-4d16-8469-66ebaeb3b906\") " Nov 01 16:02:38.244247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.244209 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-openshift-ca" pod "da4d55bc-75a1-4f9e-a45c-69bf127fddc2" (UID: "da4d55bc-75a1-4f9e-a45c-69bf127fddc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:38.244327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.244304 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-openshift-ca" pod "9d74c586-6de5-4d16-8469-66ebaeb3b906" (UID: "9d74c586-6de5-4d16-8469-66ebaeb3b906"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:38.244798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.244777 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:38.244886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.244804 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:38.246196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.246161 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-xks-ca" pod "9d74c586-6de5-4d16-8469-66ebaeb3b906" (UID: "9d74c586-6de5-4d16-8469-66ebaeb3b906"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:38.246283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.246222 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d74c586-6de5-4d16-8469-66ebaeb3b906-kube-api-access-7bhrh" pod "9d74c586-6de5-4d16-8469-66ebaeb3b906" (UID: "9d74c586-6de5-4d16-8469-66ebaeb3b906"). InnerVolumeSpecName "kube-api-access-7bhrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:38.246489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.246462 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-xks-ca" pod "da4d55bc-75a1-4f9e-a45c-69bf127fddc2" (UID: "da4d55bc-75a1-4f9e-a45c-69bf127fddc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:38.246574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.246486 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-kube-api-access-4mkzf" pod "da4d55bc-75a1-4f9e-a45c-69bf127fddc2" (UID: "da4d55bc-75a1-4f9e-a45c-69bf127fddc2"). InnerVolumeSpecName "kube-api-access-4mkzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:38.347089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.347043 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mkzf\" (UniqueName: \"kubernetes.io/projected/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-kube-api-access-4mkzf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:38.347089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.347092 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da4d55bc-75a1-4f9e-a45c-69bf127fddc2-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:38.347433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.347109 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bhrh\" (UniqueName: \"kubernetes.io/projected/9d74c586-6de5-4d16-8469-66ebaeb3b906-kube-api-access-7bhrh\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:38.347433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.347126 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d74c586-6de5-4d16-8469-66ebaeb3b906-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:38.900248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.900223 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" Nov 01 16:02:38.900648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.900220 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm" event={"ID":"da4d55bc-75a1-4f9e-a45c-69bf127fddc2","Type":"ContainerDied","Data":"80e7fc7d5e6b4f8d5fd4c4d00f3000ce4a55622807100d141f988b2c143f297c"} Nov 01 16:02:38.900648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.900344 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e7fc7d5e6b4f8d5fd4c4d00f3000ce4a55622807100d141f988b2c143f297c" Nov 01 16:02:38.902387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.902358 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" event={"ID":"9d74c586-6de5-4d16-8469-66ebaeb3b906","Type":"ContainerDied","Data":"4ae517f7844dab7446b1d1f176d83680aa60285a42b263d80a50bd58bc17e859"} Nov 01 16:02:38.902387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.902380 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2" Nov 01 16:02:38.902577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.902410 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ae517f7844dab7446b1d1f176d83680aa60285a42b263d80a50bd58bc17e859" Nov 01 16:02:38.972642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:38.972612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:39.686229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:39.686193 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:40.061025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.060950 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9"] Nov 01 16:02:40.061682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061665 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7ad054e7-5660-4df2-86ff-5fc8cd765683" containerName="revoke-keys" Nov 01 16:02:40.061733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061686 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad054e7-5660-4df2-86ff-5fc8cd765683" containerName="revoke-keys" Nov 01 16:02:40.061733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061713 4573 cpu_manager.go:414] "containerMap: removing container" podUID="da4d55bc-75a1-4f9e-a45c-69bf127fddc2" containerName="revoke-keys" Nov 01 16:02:40.061733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061721 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4d55bc-75a1-4f9e-a45c-69bf127fddc2" containerName="revoke-keys" Nov 01 16:02:40.061862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061787 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9d74c586-6de5-4d16-8469-66ebaeb3b906" containerName="revoke-keys" Nov 01 16:02:40.061862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061795 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d74c586-6de5-4d16-8469-66ebaeb3b906" containerName="revoke-keys" Nov 01 16:02:40.061959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061948 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d74c586-6de5-4d16-8469-66ebaeb3b906" containerName="revoke-keys" Nov 01 16:02:40.061998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061976 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad054e7-5660-4df2-86ff-5fc8cd765683" containerName="revoke-keys" Nov 01 16:02:40.061998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.061989 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4d55bc-75a1-4f9e-a45c-69bf127fddc2" containerName="revoke-keys" Nov 01 16:02:40.068455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.068432 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.076495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.076466 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9"] Nov 01 16:02:40.077827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.077801 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:02:40.078314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.078289 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:02:40.178646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.178612 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.178842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.178653 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.178886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.178871 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjxck\" (UniqueName: \"kubernetes.io/projected/cadea4b4-8df3-4b9f-89f2-fd9769e28187-kube-api-access-mjxck\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.280123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.280093 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjxck\" (UniqueName: \"kubernetes.io/projected/cadea4b4-8df3-4b9f-89f2-fd9769e28187-kube-api-access-mjxck\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.280328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.280234 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.280328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.280260 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.281070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.281047 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.282620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.282603 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.287950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.287932 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjxck\" (UniqueName: \"kubernetes.io/projected/cadea4b4-8df3-4b9f-89f2-fd9769e28187-kube-api-access-mjxck\") pod \"maas-api-revoke-sub-e7a8e125-4vjg9\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.392010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.391965 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:40.475968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.475933 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:40.652668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.652592 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js"] Nov 01 16:02:40.658769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.658725 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.664589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.664561 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js"] Nov 01 16:02:40.679213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.678968 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9"] Nov 01 16:02:40.787216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.787183 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.787383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.787246 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmbld\" (UniqueName: \"kubernetes.io/projected/ba172767-ecd9-4ada-9030-87c5a01e6a99-kube-api-access-jmbld\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.787452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.787394 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.888433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.888405 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.888593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.888578 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.888656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.888627 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmbld\" (UniqueName: \"kubernetes.io/projected/ba172767-ecd9-4ada-9030-87c5a01e6a99-kube-api-access-jmbld\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.889038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.889015 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.890858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.890829 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.896711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.896692 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmbld\" (UniqueName: \"kubernetes.io/projected/ba172767-ecd9-4ada-9030-87c5a01e6a99-kube-api-access-jmbld\") pod \"maas-api-revoke-sub-eceb48c4-6d6js\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:40.919253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.919196 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" event={"ID":"cadea4b4-8df3-4b9f-89f2-fd9769e28187","Type":"ContainerStarted","Data":"13acb5ff9e1e0e8f58998d8f03537d448b97d8add858accdbc453207604401ae"} Nov 01 16:02:40.983806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:40.983784 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:41.262545 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:41.262505 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba172767_ecd9_4ada_9030_87c5a01e6a99.slice/crio-4012a9d49b179067fd2a3f2327e30bc7254112e159650f1c3929a844707773f0 WatchSource:0}: Error finding container 4012a9d49b179067fd2a3f2327e30bc7254112e159650f1c3929a844707773f0: Status 404 returned error can't find the container with id 4012a9d49b179067fd2a3f2327e30bc7254112e159650f1c3929a844707773f0 Nov 01 16:02:41.262902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:41.262720 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js"] Nov 01 16:02:41.284787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:41.284736 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:41.935153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:41.935109 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" event={"ID":"ba172767-ecd9-4ada-9030-87c5a01e6a99","Type":"ContainerStarted","Data":"74904f30b3a95207e8a3ec79eae0ee8f82f302d92499113b6580d35914c291d9"} Nov 01 16:02:41.935345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:41.935208 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" event={"ID":"ba172767-ecd9-4ada-9030-87c5a01e6a99","Type":"ContainerStarted","Data":"4012a9d49b179067fd2a3f2327e30bc7254112e159650f1c3929a844707773f0"} Nov 01 16:02:41.936729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:41.936704 4573 generic.go:356] "Generic (PLEG): container finished" podID="cadea4b4-8df3-4b9f-89f2-fd9769e28187" containerID="6f1a18ec2ab4bb7117b5798feef323c7f959e3733abe26ba9277ffca03c6a842" exitCode=0 Nov 01 16:02:41.936822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:41.936782 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" event={"ID":"cadea4b4-8df3-4b9f-89f2-fd9769e28187","Type":"ContainerDied","Data":"6f1a18ec2ab4bb7117b5798feef323c7f959e3733abe26ba9277ffca03c6a842"} Nov 01 16:02:41.953485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:41.953444 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" podStartSLOduration=1.953430668 podStartE2EDuration="1.953430668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:41.951945998 +0000 UTC m=+899.655111935" watchObservedRunningTime="2026-11-01 16:02:41.953430668 +0000 UTC m=+899.656596604" Nov 01 16:02:41.978305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:41.978279 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:42.563845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:42.563769 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:02:42.567518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:42.567496 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:02:42.575712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:42.575684 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:02:42.579342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:42.579323 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:02:42.670400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:42.670364 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:42.945342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:42.945309 4573 generic.go:356] "Generic (PLEG): container finished" podID="ba172767-ecd9-4ada-9030-87c5a01e6a99" containerID="74904f30b3a95207e8a3ec79eae0ee8f82f302d92499113b6580d35914c291d9" exitCode=0 Nov 01 16:02:42.945501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:42.945376 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" event={"ID":"ba172767-ecd9-4ada-9030-87c5a01e6a99","Type":"ContainerDied","Data":"74904f30b3a95207e8a3ec79eae0ee8f82f302d92499113b6580d35914c291d9"} Nov 01 16:02:43.124845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.124820 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:43.224468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.224382 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadea4b4-8df3-4b9f-89f2-fd9769e28187-kube-api-access-mjxck\" (UniqueName: \"kubernetes.io/projected/cadea4b4-8df3-4b9f-89f2-fd9769e28187-kube-api-access-mjxck\") pod \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " Nov 01 16:02:43.224468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.224457 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-openshift-ca\") pod \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " Nov 01 16:02:43.224664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.224507 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-xks-ca\") pod \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\" (UID: \"cadea4b4-8df3-4b9f-89f2-fd9769e28187\") " Nov 01 16:02:43.224847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.224823 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-openshift-ca" pod "cadea4b4-8df3-4b9f-89f2-fd9769e28187" (UID: "cadea4b4-8df3-4b9f-89f2-fd9769e28187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:43.225299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.225279 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:43.226631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.226601 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-xks-ca" pod "cadea4b4-8df3-4b9f-89f2-fd9769e28187" (UID: "cadea4b4-8df3-4b9f-89f2-fd9769e28187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:43.226720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.226681 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadea4b4-8df3-4b9f-89f2-fd9769e28187-kube-api-access-mjxck" pod "cadea4b4-8df3-4b9f-89f2-fd9769e28187" (UID: "cadea4b4-8df3-4b9f-89f2-fd9769e28187"). InnerVolumeSpecName "kube-api-access-mjxck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:43.327287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.327253 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjxck\" (UniqueName: \"kubernetes.io/projected/cadea4b4-8df3-4b9f-89f2-fd9769e28187-kube-api-access-mjxck\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:43.327287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.327286 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadea4b4-8df3-4b9f-89f2-fd9769e28187-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:43.366047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.366012 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:43.824793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.824744 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b"] Nov 01 16:02:43.825440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.825425 4573 cpu_manager.go:414] "containerMap: removing container" podUID="cadea4b4-8df3-4b9f-89f2-fd9769e28187" containerName="revoke-keys" Nov 01 16:02:43.825484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.825441 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadea4b4-8df3-4b9f-89f2-fd9769e28187" containerName="revoke-keys" Nov 01 16:02:43.825716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.825700 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadea4b4-8df3-4b9f-89f2-fd9769e28187" containerName="revoke-keys" Nov 01 16:02:43.829864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.829843 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.841473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.841448 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b"] Nov 01 16:02:43.860554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.859763 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn"] Nov 01 16:02:43.867847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.867680 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.873959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.873933 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn"] Nov 01 16:02:43.954660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.954632 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" Nov 01 16:02:43.954660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.954655 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9" event={"ID":"cadea4b4-8df3-4b9f-89f2-fd9769e28187","Type":"ContainerDied","Data":"13acb5ff9e1e0e8f58998d8f03537d448b97d8add858accdbc453207604401ae"} Nov 01 16:02:43.954884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.954701 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13acb5ff9e1e0e8f58998d8f03537d448b97d8add858accdbc453207604401ae" Nov 01 16:02:43.963979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.963949 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.964115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.963990 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7lc6\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-kube-api-access-q7lc6\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.964115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964058 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.964115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964112 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964135 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cct\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-kube-api-access-69cct\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964252 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.964287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964286 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964316 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.964451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964370 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964406 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964430 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964508 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964561 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964644 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.964857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964664 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.964857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964730 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:43.964857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964789 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:43.964857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:43.964807 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.023514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.023483 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq"] Nov 01 16:02:44.030199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.030103 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.043835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.043807 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq"] Nov 01 16:02:44.066510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066477 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.066670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066523 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.066670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066562 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.066670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066642 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.066940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066721 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.066940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066806 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.066940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066839 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.066940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066917 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.067211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066965 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.067211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.066996 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.067211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.067040 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.067211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.067065 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7lc6\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-kube-api-access-q7lc6\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.067211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.067093 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.067211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.067162 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.067211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.067192 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cct\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-kube-api-access-69cct\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.067665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.067261 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.067665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.067302 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.067665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.067346 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.068037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.068015 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.068436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.068247 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.068553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.068494 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.068553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.068487 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.068745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.068721 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.069572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.069540 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.070421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.070396 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-certs\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.071095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.071068 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istiod-ca-cert\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.071469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.071314 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-data\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.071469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.071410 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-credential-socket\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.072171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.072144 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.072350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.072332 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.074533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.074511 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-envoy\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.078122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.078066 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7lc6\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-kube-api-access-q7lc6\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.078366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.078346 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cct\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-kube-api-access-69cct\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.079033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.079011 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.081319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.081295 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-podinfo\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.084973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.084949 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-token\") pod \"e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.135319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.135297 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:44.147617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.147596 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:44.168936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.168908 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.169088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.169060 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnzjr\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-kube-api-access-gnzjr\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.169151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.169106 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.169206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.169151 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.169261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.169243 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.169316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.169277 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.169316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.169303 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.169451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.169433 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.169839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.169807 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.187789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.187566 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.271311 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-openshift-ca\") pod \"ba172767-ecd9-4ada-9030-87c5a01e6a99\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.271452 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba172767-ecd9-4ada-9030-87c5a01e6a99-kube-api-access-jmbld\" (UniqueName: \"kubernetes.io/projected/ba172767-ecd9-4ada-9030-87c5a01e6a99-kube-api-access-jmbld\") pod \"ba172767-ecd9-4ada-9030-87c5a01e6a99\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.271488 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-xks-ca\") pod \"ba172767-ecd9-4ada-9030-87c5a01e6a99\" (UID: \"ba172767-ecd9-4ada-9030-87c5a01e6a99\") " Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.271899 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnzjr\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-kube-api-access-gnzjr\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.271956 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.272003 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.272097 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.272130 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.272154 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.272295 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.272380 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.272429 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.275709 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.276074 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istiod-ca-cert\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.276089 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-credential-socket\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.276498 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-openshift-ca" pod "ba172767-ecd9-4ada-9030-87c5a01e6a99" (UID: "ba172767-ecd9-4ada-9030-87c5a01e6a99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:44.277802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.276676 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-certs\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.280337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.277946 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-data\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.286941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.286895 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-token\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.289273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.289141 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnzjr\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-kube-api-access-gnzjr\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.297157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.297123 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba172767-ecd9-4ada-9030-87c5a01e6a99-kube-api-access-jmbld" pod "ba172767-ecd9-4ada-9030-87c5a01e6a99" (UID: "ba172767-ecd9-4ada-9030-87c5a01e6a99"). InnerVolumeSpecName "kube-api-access-jmbld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:44.297303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.297278 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-xks-ca" pod "ba172767-ecd9-4ada-9030-87c5a01e6a99" (UID: "ba172767-ecd9-4ada-9030-87c5a01e6a99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:44.297466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.297439 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-envoy\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.299836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.299763 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-podinfo\") pod \"e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.334313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.334210 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:44.358455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.358426 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs"] Nov 01 16:02:44.359120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.359101 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ba172767-ecd9-4ada-9030-87c5a01e6a99" containerName="revoke-keys" Nov 01 16:02:44.359120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.359117 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba172767-ecd9-4ada-9030-87c5a01e6a99" containerName="revoke-keys" Nov 01 16:02:44.359281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.359242 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba172767-ecd9-4ada-9030-87c5a01e6a99" containerName="revoke-keys" Nov 01 16:02:44.370465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.370440 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.371469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.371443 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs"] Nov 01 16:02:44.374979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.374934 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:44.375135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.375097 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmbld\" (UniqueName: \"kubernetes.io/projected/ba172767-ecd9-4ada-9030-87c5a01e6a99-kube-api-access-jmbld\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:44.375135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.375117 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba172767-ecd9-4ada-9030-87c5a01e6a99-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:44.408705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.408421 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:44.471195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.471156 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.471316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.471236 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.471316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.471279 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.471416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.471382 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b"] Nov 01 16:02:44.477198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.476791 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvkr\" (UniqueName: \"kubernetes.io/projected/63028bf5-b600-4781-9dbc-dde2cfb8917f-kube-api-access-smvkr\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.477198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.476955 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.477198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.477026 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.537049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.536673 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn"] Nov 01 16:02:44.539473 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:44.539213 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode763fd9f_5b6f_4086_83c3_8b5f4a0de00b.slice/crio-a821be868061cf361891a4d418f276f73c1b2643794f07210abbe2c22019cdfd WatchSource:0}: Error finding container a821be868061cf361891a4d418f276f73c1b2643794f07210abbe2c22019cdfd: Status 404 returned error can't find the container with id a821be868061cf361891a4d418f276f73c1b2643794f07210abbe2c22019cdfd Nov 01 16:02:44.553283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.553244 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.553498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.553472 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.553654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.553634 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.579269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.579237 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvkr\" (UniqueName: \"kubernetes.io/projected/63028bf5-b600-4781-9dbc-dde2cfb8917f-kube-api-access-smvkr\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.580115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.580094 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.580915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.580893 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.581500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.581471 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.583134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.583113 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.587673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.587650 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvkr\" (UniqueName: \"kubernetes.io/projected/63028bf5-b600-4781-9dbc-dde2cfb8917f-kube-api-access-smvkr\") pod \"maas-api-revoke-sub-bf0cd681-stjrs\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.739332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.739299 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:44.774951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.772535 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq"] Nov 01 16:02:44.776883 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:44.776856 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7238dde2_4482_4ff0_bafe_f7be1b2434c0.slice/crio-79ce87922357cb95e581cb9a2f768879d18268059a740b99648fb541b88906ee WatchSource:0}: Error finding container 79ce87922357cb95e581cb9a2f768879d18268059a740b99648fb541b88906ee: Status 404 returned error can't find the container with id 79ce87922357cb95e581cb9a2f768879d18268059a740b99648fb541b88906ee Nov 01 16:02:44.779699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.779667 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.779793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.779734 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.779793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.779778 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:02:44.983962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.983926 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" event={"ID":"ba172767-ecd9-4ada-9030-87c5a01e6a99","Type":"ContainerDied","Data":"4012a9d49b179067fd2a3f2327e30bc7254112e159650f1c3929a844707773f0"} Nov 01 16:02:44.985125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.985106 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4012a9d49b179067fd2a3f2327e30bc7254112e159650f1c3929a844707773f0" Nov 01 16:02:44.985811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.984540 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js" Nov 01 16:02:44.991724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.991627 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" event={"ID":"c2b0a1ec-5862-4872-87cc-c6d5b040053c","Type":"ContainerStarted","Data":"458e5d5b0e16192d433791b995be3e67b0a670f03dc195104d512a87c7e8e75a"} Nov 01 16:02:44.991724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.991696 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" event={"ID":"c2b0a1ec-5862-4872-87cc-c6d5b040053c","Type":"ContainerStarted","Data":"b0c7562632797332df8b066257851784ffdd6a12461aa1f8b10bbfda25e73a86"} Nov 01 16:02:44.997594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.997136 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" event={"ID":"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b","Type":"ContainerStarted","Data":"2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa"} Nov 01 16:02:44.997594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:44.997201 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" event={"ID":"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b","Type":"ContainerStarted","Data":"a821be868061cf361891a4d418f276f73c1b2643794f07210abbe2c22019cdfd"} Nov 01 16:02:45.011413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.011378 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" event={"ID":"7238dde2-4482-4ff0-bafe-f7be1b2434c0","Type":"ContainerStarted","Data":"58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026"} Nov 01 16:02:45.011643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.011627 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" event={"ID":"7238dde2-4482-4ff0-bafe-f7be1b2434c0","Type":"ContainerStarted","Data":"79ce87922357cb95e581cb9a2f768879d18268059a740b99648fb541b88906ee"} Nov 01 16:02:45.054638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.054570 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podStartSLOduration=2.054536338 podStartE2EDuration="2.054536338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:45.010707876 +0000 UTC m=+902.713873815" watchObservedRunningTime="2026-11-01 16:02:45.054536338 +0000 UTC m=+902.757702275" Nov 01 16:02:45.056521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.056478 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podStartSLOduration=2.05646344 podStartE2EDuration="2.05646344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:45.042706665 +0000 UTC m=+902.745872601" watchObservedRunningTime="2026-11-01 16:02:45.05646344 +0000 UTC m=+902.759629376" Nov 01 16:02:45.090154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.090086 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs"] Nov 01 16:02:45.090686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.090641 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podStartSLOduration=2.090622982 podStartE2EDuration="2.090622982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:45.076876326 +0000 UTC m=+902.780042262" watchObservedRunningTime="2026-11-01 16:02:45.090622982 +0000 UTC m=+902.793788920" Nov 01 16:02:45.153864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.153151 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:45.153864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.153592 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:02:45.153864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.153643 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 16:02:45.218614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.218379 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:45.220166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.219846 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 16:02:45.220166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.219904 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 16:02:45.276175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.275955 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:45.421533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.421147 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:45.421533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.421260 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 16:02:45.421533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:45.421302 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 16:02:46.054492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.054404 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" event={"ID":"63028bf5-b600-4781-9dbc-dde2cfb8917f","Type":"ContainerStarted","Data":"3e64c57bc5a7be75e8e84bf84041edf4df800702a2e2f6a457801f0f9a40b9ae"} Nov 01 16:02:46.054492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.054470 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" event={"ID":"63028bf5-b600-4781-9dbc-dde2cfb8917f","Type":"ContainerStarted","Data":"36ac9717ea897b27820c018ce2d4ec3b4a061905947a99e5b5bc2de419c3e792"} Nov 01 16:02:46.092776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.091837 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" podStartSLOduration=2.091816273 podStartE2EDuration="2.091816273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:46.090161939 +0000 UTC m=+903.793327876" watchObservedRunningTime="2026-11-01 16:02:46.091816273 +0000 UTC m=+903.794982211" Nov 01 16:02:46.148924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.148890 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:02:46.149170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.149143 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 16:02:46.208322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.195388 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 16:02:46.208322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.195453 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 16:02:46.382333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.382230 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:46.428737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.428263 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 16:02:46.428737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:46.428328 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 16:02:47.138668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.138626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:47.147839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.140975 4573 generic.go:356] "Generic (PLEG): container finished" podID="63028bf5-b600-4781-9dbc-dde2cfb8917f" containerID="3e64c57bc5a7be75e8e84bf84041edf4df800702a2e2f6a457801f0f9a40b9ae" exitCode=0 Nov 01 16:02:47.147839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.141789 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" event={"ID":"63028bf5-b600-4781-9dbc-dde2cfb8917f","Type":"ContainerDied","Data":"3e64c57bc5a7be75e8e84bf84041edf4df800702a2e2f6a457801f0f9a40b9ae"} Nov 01 16:02:47.150791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.149395 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:02:47.150791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.149450 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 16:02:47.191012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.190345 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 16:02:47.191012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.190405 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 16:02:47.411712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.411614 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 16:02:47.411984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:47.411940 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 16:02:48.047990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.047571 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:48.173774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.172157 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:02:48.173774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.172222 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 16:02:48.199825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.194774 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 16:02:48.199825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.194832 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 16:02:48.423265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.422815 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 16:02:48.423265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.422879 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 16:02:48.736840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.736727 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7"] Nov 01 16:02:48.896083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.882865 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7"] Nov 01 16:02:48.896083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.883035 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:48.946847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.946353 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:48.946847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.946466 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjcdz\" (UniqueName: \"kubernetes.io/projected/9d2067eb-c85b-4c6b-866f-339bcd750044-kube-api-access-kjcdz\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:48.946847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.946543 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:49.013838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:48.990013 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:49.036788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.025040 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:49.050410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.050372 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-openshift-ca\") pod \"63028bf5-b600-4781-9dbc-dde2cfb8917f\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " Nov 01 16:02:49.050589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.050478 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63028bf5-b600-4781-9dbc-dde2cfb8917f-kube-api-access-smvkr\" (UniqueName: \"kubernetes.io/projected/63028bf5-b600-4781-9dbc-dde2cfb8917f-kube-api-access-smvkr\") pod \"63028bf5-b600-4781-9dbc-dde2cfb8917f\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " Nov 01 16:02:49.050589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.050528 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-xks-ca\") pod \"63028bf5-b600-4781-9dbc-dde2cfb8917f\" (UID: \"63028bf5-b600-4781-9dbc-dde2cfb8917f\") " Nov 01 16:02:49.050936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.050910 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:49.051045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.051033 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjcdz\" (UniqueName: \"kubernetes.io/projected/9d2067eb-c85b-4c6b-866f-339bcd750044-kube-api-access-kjcdz\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:49.052296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.051124 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:49.053625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.052470 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-openshift-ca" pod "63028bf5-b600-4781-9dbc-dde2cfb8917f" (UID: "63028bf5-b600-4781-9dbc-dde2cfb8917f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:49.053625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.053350 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:49.055274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.055250 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-xks-ca" pod "63028bf5-b600-4781-9dbc-dde2cfb8917f" (UID: "63028bf5-b600-4781-9dbc-dde2cfb8917f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:49.055411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.055389 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:49.230776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.209152 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63028bf5-b600-4781-9dbc-dde2cfb8917f-kube-api-access-smvkr" pod "63028bf5-b600-4781-9dbc-dde2cfb8917f" (UID: "63028bf5-b600-4781-9dbc-dde2cfb8917f"). InnerVolumeSpecName "kube-api-access-smvkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:49.256152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.255499 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjcdz\" (UniqueName: \"kubernetes.io/projected/9d2067eb-c85b-4c6b-866f-339bcd750044-kube-api-access-kjcdz\") pod \"maas-api-revoke-sub-3b46e218-bw9c7\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:49.295866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.295688 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:49.296141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.296082 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smvkr\" (UniqueName: \"kubernetes.io/projected/63028bf5-b600-4781-9dbc-dde2cfb8917f-kube-api-access-smvkr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:49.296141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.296112 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63028bf5-b600-4781-9dbc-dde2cfb8917f-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:49.323803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.303244 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:49.426795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.423140 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 16:02:49.426795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.423200 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 16:02:49.461784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.455411 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:02:49.461784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.455499 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 16:02:49.505786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.505326 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s"] Nov 01 16:02:49.553888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.506696 4573 cpu_manager.go:414] "containerMap: removing container" podUID="63028bf5-b600-4781-9dbc-dde2cfb8917f" containerName="revoke-keys" Nov 01 16:02:49.553888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.506726 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63028bf5-b600-4781-9dbc-dde2cfb8917f" containerName="revoke-keys" Nov 01 16:02:49.553888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.506949 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="63028bf5-b600-4781-9dbc-dde2cfb8917f" containerName="revoke-keys" Nov 01 16:02:49.556093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.555497 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s"] Nov 01 16:02:49.556093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.555614 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.571299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.561159 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" event={"ID":"63028bf5-b600-4781-9dbc-dde2cfb8917f","Type":"ContainerDied","Data":"36ac9717ea897b27820c018ce2d4ec3b4a061905947a99e5b5bc2de419c3e792"} Nov 01 16:02:49.571299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.561229 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36ac9717ea897b27820c018ce2d4ec3b4a061905947a99e5b5bc2de419c3e792" Nov 01 16:02:49.571299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.561298 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs" Nov 01 16:02:49.689163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.660972 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.689163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.661118 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.689163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.661187 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbpcc\" (UniqueName: \"kubernetes.io/projected/d797d9a4-54a5-479b-a186-8d8d21b99f58-kube-api-access-xbpcc\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.722338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.722288 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 16:02:49.722627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.722597 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 16:02:49.780775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.778942 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.780775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.779084 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbpcc\" (UniqueName: \"kubernetes.io/projected/d797d9a4-54a5-479b-a186-8d8d21b99f58-kube-api-access-xbpcc\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.780775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.779397 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.781075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.780830 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.816829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.801476 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.844919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.844074 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbpcc\" (UniqueName: \"kubernetes.io/projected/d797d9a4-54a5-479b-a186-8d8d21b99f58-kube-api-access-xbpcc\") pod \"maas-api-revoke-sub-5bc25d5a-9m85s\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:49.851620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:49.851589 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:50.060776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.060401 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:50.257049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.255953 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:02:50.257049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.256009 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 16:02:50.257049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.256772 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 16:02:50.257049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.256814 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 16:02:50.419774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.415565 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 16:02:50.419774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.415658 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 16:02:50.557286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.556949 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7"] Nov 01 16:02:50.559328 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:50.559299 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2067eb_c85b_4c6b_866f_339bcd750044.slice/crio-58d7110728a3cd9312371166c1ef5fc95f19c25ca06317681ac9e938fa1ee429 WatchSource:0}: Error finding container 58d7110728a3cd9312371166c1ef5fc95f19c25ca06317681ac9e938fa1ee429: Status 404 returned error can't find the container with id 58d7110728a3cd9312371166c1ef5fc95f19c25ca06317681ac9e938fa1ee429 Nov 01 16:02:50.624165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.624085 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:50.922836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:50.911794 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" event={"ID":"9d2067eb-c85b-4c6b-866f-339bcd750044","Type":"ContainerStarted","Data":"58d7110728a3cd9312371166c1ef5fc95f19c25ca06317681ac9e938fa1ee429"} Nov 01 16:02:51.151396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.150828 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:02:51.151396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.150892 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 16:02:51.154024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.153997 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s"] Nov 01 16:02:51.203777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.192828 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 16:02:51.203777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.192908 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 16:02:51.296775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.296172 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:51.414789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.411600 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 16:02:51.414789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.411681 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 16:02:51.927364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.927324 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" event={"ID":"d797d9a4-54a5-479b-a186-8d8d21b99f58","Type":"ContainerStarted","Data":"cf54da6f3816cb3ea902833ad2478289b14fa0287e98f872e806132845e6ef5f"} Nov 01 16:02:51.927525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.927419 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" event={"ID":"d797d9a4-54a5-479b-a186-8d8d21b99f58","Type":"ContainerStarted","Data":"3089cc981b19923d3b49203752d9b24a56ba1e741937b501678856f7c1c98c20"} Nov 01 16:02:51.929523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.929495 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" event={"ID":"9d2067eb-c85b-4c6b-866f-339bcd750044","Type":"ContainerStarted","Data":"047123709b07131b916974dac91c190f175fb91d8a7b27cdc1cf5dd970904bd9"} Nov 01 16:02:51.948643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.948593 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" podStartSLOduration=2.9485759209999998 podStartE2EDuration="2.948575921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:51.941962209 +0000 UTC m=+909.645128158" watchObservedRunningTime="2026-11-01 16:02:51.948575921 +0000 UTC m=+909.651741857" Nov 01 16:02:51.961990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:51.961904 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" podStartSLOduration=3.961886765 podStartE2EDuration="3.961886765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:51.958386761 +0000 UTC m=+909.661552733" watchObservedRunningTime="2026-11-01 16:02:51.961886765 +0000 UTC m=+909.665052702" Nov 01 16:02:52.075869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.075790 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:52.161774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.156156 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 16:02:52.161774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.156223 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 16:02:52.197535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.188275 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 16:02:52.197535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.188334 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 16:02:52.413588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.413547 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 01 16:02:52.414093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.413605 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 01 16:02:52.802829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.802392 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78"] Nov 01 16:02:52.835775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.827843 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78"] Nov 01 16:02:52.835775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.827968 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:52.900780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.899448 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c5wb\" (UniqueName: \"kubernetes.io/projected/a526e313-2c7d-485e-82b4-91b1e19016b7-kube-api-access-4c5wb\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:52.900780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.899556 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:52.900780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.899624 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:52.933321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.933267 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:52.953091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.952571 4573 generic.go:356] "Generic (PLEG): container finished" podID="9d2067eb-c85b-4c6b-866f-339bcd750044" containerID="047123709b07131b916974dac91c190f175fb91d8a7b27cdc1cf5dd970904bd9" exitCode=0 Nov 01 16:02:52.953832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.953398 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" event={"ID":"9d2067eb-c85b-4c6b-866f-339bcd750044","Type":"ContainerDied","Data":"047123709b07131b916974dac91c190f175fb91d8a7b27cdc1cf5dd970904bd9"} Nov 01 16:02:52.959360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.959338 4573 generic.go:356] "Generic (PLEG): container finished" podID="d797d9a4-54a5-479b-a186-8d8d21b99f58" containerID="cf54da6f3816cb3ea902833ad2478289b14fa0287e98f872e806132845e6ef5f" exitCode=0 Nov 01 16:02:52.960380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.959493 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" event={"ID":"d797d9a4-54a5-479b-a186-8d8d21b99f58","Type":"ContainerDied","Data":"cf54da6f3816cb3ea902833ad2478289b14fa0287e98f872e806132845e6ef5f"} Nov 01 16:02:52.974411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.974386 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk"] Nov 01 16:02:52.993144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:52.993117 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.047027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.046546 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c5wb\" (UniqueName: \"kubernetes.io/projected/a526e313-2c7d-485e-82b4-91b1e19016b7-kube-api-access-4c5wb\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:53.047027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.046697 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:53.065777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.057517 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:53.065777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.058729 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:53.097075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.096904 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c5wb\" (UniqueName: \"kubernetes.io/projected/a526e313-2c7d-485e-82b4-91b1e19016b7-kube-api-access-4c5wb\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:53.097075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.097002 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk"] Nov 01 16:02:53.103275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.103216 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-708957cd-rnk78\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:53.163215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.163183 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:53.165057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.165035 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:53.170741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.170720 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:02:53.171113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.171092 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:53.173341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.173314 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.173646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.173625 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frcw8\" (UniqueName: \"kubernetes.io/projected/cfa52ad9-568f-438f-8f40-4748c2444159-kube-api-access-frcw8\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.173889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.173868 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.200989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.198699 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:53.200989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.199251 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:53.200989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.199890 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:02:53.290105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.288910 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.290105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.289148 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frcw8\" (UniqueName: \"kubernetes.io/projected/cfa52ad9-568f-438f-8f40-4748c2444159-kube-api-access-frcw8\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.290105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.289244 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.292420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.292381 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.299947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.299916 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.315061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.315026 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frcw8\" (UniqueName: \"kubernetes.io/projected/cfa52ad9-568f-438f-8f40-4748c2444159-kube-api-access-frcw8\") pod \"maas-api-revoke-sub-6bb22d5f-54glk\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.417580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.417549 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:53.419009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.418980 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:53.419704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.419684 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:02:53.481110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.480829 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:53.512691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.512660 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78"] Nov 01 16:02:53.740076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.740039 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:53.789413 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:53.789381 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa52ad9_568f_438f_8f40_4748c2444159.slice/crio-3b2a1cb4bb363018a9c029aa5d6b2ec856675319bbb0b02681aaa68ed691c29c WatchSource:0}: Error finding container 3b2a1cb4bb363018a9c029aa5d6b2ec856675319bbb0b02681aaa68ed691c29c: Status 404 returned error can't find the container with id 3b2a1cb4bb363018a9c029aa5d6b2ec856675319bbb0b02681aaa68ed691c29c Nov 01 16:02:53.789539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.789491 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk"] Nov 01 16:02:53.970151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.970067 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" event={"ID":"cfa52ad9-568f-438f-8f40-4748c2444159","Type":"ContainerStarted","Data":"3b2a1cb4bb363018a9c029aa5d6b2ec856675319bbb0b02681aaa68ed691c29c"} Nov 01 16:02:53.971841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.971801 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" event={"ID":"a526e313-2c7d-485e-82b4-91b1e19016b7","Type":"ContainerStarted","Data":"47d202bb43e3b3b6903d0689ecf5c3919d3f06869f603df5a8fc2556f3658c87"} Nov 01 16:02:53.971956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.971854 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" event={"ID":"a526e313-2c7d-485e-82b4-91b1e19016b7","Type":"ContainerStarted","Data":"7f000eca75905c7e8f8e75592bf7b66bcdfd13853caf4abeb7905b46dba34977"} Nov 01 16:02:53.987180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:53.987133 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" podStartSLOduration=1.9871195579999998 podStartE2EDuration="1.987119558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:53.985604058 +0000 UTC m=+911.688769994" watchObservedRunningTime="2026-11-01 16:02:53.987119558 +0000 UTC m=+911.690285493" Nov 01 16:02:54.137576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.137547 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:54.192324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.192298 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:54.251590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.251511 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-openshift-ca\") pod \"9d2067eb-c85b-4c6b-866f-339bcd750044\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " Nov 01 16:02:54.251740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.251591 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d2067eb-c85b-4c6b-866f-339bcd750044-kube-api-access-kjcdz\" (UniqueName: \"kubernetes.io/projected/9d2067eb-c85b-4c6b-866f-339bcd750044-kube-api-access-kjcdz\") pod \"9d2067eb-c85b-4c6b-866f-339bcd750044\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " Nov 01 16:02:54.251840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.251820 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-xks-ca\") pod \"9d2067eb-c85b-4c6b-866f-339bcd750044\" (UID: \"9d2067eb-c85b-4c6b-866f-339bcd750044\") " Nov 01 16:02:54.251983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.251953 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-openshift-ca" pod "9d2067eb-c85b-4c6b-866f-339bcd750044" (UID: "9d2067eb-c85b-4c6b-866f-339bcd750044"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:54.252664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.252623 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:54.253825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.253797 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2067eb-c85b-4c6b-866f-339bcd750044-kube-api-access-kjcdz" pod "9d2067eb-c85b-4c6b-866f-339bcd750044" (UID: "9d2067eb-c85b-4c6b-866f-339bcd750044"). InnerVolumeSpecName "kube-api-access-kjcdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:54.254345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.254323 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-xks-ca" pod "9d2067eb-c85b-4c6b-866f-339bcd750044" (UID: "9d2067eb-c85b-4c6b-866f-339bcd750044"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:54.354095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.354064 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-openshift-ca\") pod \"d797d9a4-54a5-479b-a186-8d8d21b99f58\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " Nov 01 16:02:54.354371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.354354 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-xks-ca\") pod \"d797d9a4-54a5-479b-a186-8d8d21b99f58\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " Nov 01 16:02:54.354445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.354416 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-openshift-ca" pod "d797d9a4-54a5-479b-a186-8d8d21b99f58" (UID: "d797d9a4-54a5-479b-a186-8d8d21b99f58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:54.354504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.354450 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d797d9a4-54a5-479b-a186-8d8d21b99f58-kube-api-access-xbpcc\" (UniqueName: \"kubernetes.io/projected/d797d9a4-54a5-479b-a186-8d8d21b99f58-kube-api-access-xbpcc\") pod \"d797d9a4-54a5-479b-a186-8d8d21b99f58\" (UID: \"d797d9a4-54a5-479b-a186-8d8d21b99f58\") " Nov 01 16:02:54.355037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.355013 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjcdz\" (UniqueName: \"kubernetes.io/projected/9d2067eb-c85b-4c6b-866f-339bcd750044-kube-api-access-kjcdz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:54.355037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.355036 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d2067eb-c85b-4c6b-866f-339bcd750044-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:54.355240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.355045 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:54.356434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.356411 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-xks-ca" pod "d797d9a4-54a5-479b-a186-8d8d21b99f58" (UID: "d797d9a4-54a5-479b-a186-8d8d21b99f58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:54.356864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.356847 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d797d9a4-54a5-479b-a186-8d8d21b99f58-kube-api-access-xbpcc" pod "d797d9a4-54a5-479b-a186-8d8d21b99f58" (UID: "d797d9a4-54a5-479b-a186-8d8d21b99f58"). InnerVolumeSpecName "kube-api-access-xbpcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:54.366688 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:02:54.366665 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:02:54.456832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.456810 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d797d9a4-54a5-479b-a186-8d8d21b99f58-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:54.456832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.456831 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbpcc\" (UniqueName: \"kubernetes.io/projected/d797d9a4-54a5-479b-a186-8d8d21b99f58-kube-api-access-xbpcc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:54.552803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.552706 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:54.982371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.982344 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" Nov 01 16:02:54.982562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.982348 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7" event={"ID":"9d2067eb-c85b-4c6b-866f-339bcd750044","Type":"ContainerDied","Data":"58d7110728a3cd9312371166c1ef5fc95f19c25ca06317681ac9e938fa1ee429"} Nov 01 16:02:54.982562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.982485 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58d7110728a3cd9312371166c1ef5fc95f19c25ca06317681ac9e938fa1ee429" Nov 01 16:02:54.984917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.984212 4573 generic.go:356] "Generic (PLEG): container finished" podID="cfa52ad9-568f-438f-8f40-4748c2444159" containerID="babacd3c6b5798cddcaedb8fc0924001fdebe8c3abff58975b6da2c9593530c7" exitCode=0 Nov 01 16:02:54.984917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.984309 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" event={"ID":"cfa52ad9-568f-438f-8f40-4748c2444159","Type":"ContainerDied","Data":"babacd3c6b5798cddcaedb8fc0924001fdebe8c3abff58975b6da2c9593530c7"} Nov 01 16:02:54.985861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.985847 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" Nov 01 16:02:54.985918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.985858 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s" event={"ID":"d797d9a4-54a5-479b-a186-8d8d21b99f58","Type":"ContainerDied","Data":"3089cc981b19923d3b49203752d9b24a56ba1e741937b501678856f7c1c98c20"} Nov 01 16:02:54.985918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.985895 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3089cc981b19923d3b49203752d9b24a56ba1e741937b501678856f7c1c98c20" Nov 01 16:02:54.987827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.987800 4573 generic.go:356] "Generic (PLEG): container finished" podID="a526e313-2c7d-485e-82b4-91b1e19016b7" containerID="47d202bb43e3b3b6903d0689ecf5c3919d3f06869f603df5a8fc2556f3658c87" exitCode=0 Nov 01 16:02:54.987917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:54.987886 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" event={"ID":"a526e313-2c7d-485e-82b4-91b1e19016b7","Type":"ContainerDied","Data":"47d202bb43e3b3b6903d0689ecf5c3919d3f06869f603df5a8fc2556f3658c87"} Nov 01 16:02:55.560077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:55.560038 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:56.228074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.228049 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:56.231613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.231593 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:56.375298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.375268 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:56.386645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.386618 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-xks-ca\") pod \"cfa52ad9-568f-438f-8f40-4748c2444159\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " Nov 01 16:02:56.386815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.386698 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-xks-ca\") pod \"a526e313-2c7d-485e-82b4-91b1e19016b7\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " Nov 01 16:02:56.386878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.386858 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-openshift-ca\") pod \"cfa52ad9-568f-438f-8f40-4748c2444159\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " Nov 01 16:02:56.386974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.386960 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa52ad9-568f-438f-8f40-4748c2444159-kube-api-access-frcw8\" (UniqueName: \"kubernetes.io/projected/cfa52ad9-568f-438f-8f40-4748c2444159-kube-api-access-frcw8\") pod \"cfa52ad9-568f-438f-8f40-4748c2444159\" (UID: \"cfa52ad9-568f-438f-8f40-4748c2444159\") " Nov 01 16:02:56.387080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.387056 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a526e313-2c7d-485e-82b4-91b1e19016b7-kube-api-access-4c5wb\" (UniqueName: \"kubernetes.io/projected/a526e313-2c7d-485e-82b4-91b1e19016b7-kube-api-access-4c5wb\") pod \"a526e313-2c7d-485e-82b4-91b1e19016b7\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " Nov 01 16:02:56.387202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.387178 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-openshift-ca\") pod \"a526e313-2c7d-485e-82b4-91b1e19016b7\" (UID: \"a526e313-2c7d-485e-82b4-91b1e19016b7\") " Nov 01 16:02:56.387618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.387595 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-openshift-ca" pod "a526e313-2c7d-485e-82b4-91b1e19016b7" (UID: "a526e313-2c7d-485e-82b4-91b1e19016b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:56.387717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.387259 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-openshift-ca" pod "cfa52ad9-568f-438f-8f40-4748c2444159" (UID: "cfa52ad9-568f-438f-8f40-4748c2444159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:02:56.388007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.387990 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:56.388089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.388012 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:56.388954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.388924 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-xks-ca" pod "cfa52ad9-568f-438f-8f40-4748c2444159" (UID: "cfa52ad9-568f-438f-8f40-4748c2444159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:56.389043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.388980 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-xks-ca" pod "a526e313-2c7d-485e-82b4-91b1e19016b7" (UID: "a526e313-2c7d-485e-82b4-91b1e19016b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:02:56.389387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.389370 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa52ad9-568f-438f-8f40-4748c2444159-kube-api-access-frcw8" pod "cfa52ad9-568f-438f-8f40-4748c2444159" (UID: "cfa52ad9-568f-438f-8f40-4748c2444159"). InnerVolumeSpecName "kube-api-access-frcw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:56.389444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.389395 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a526e313-2c7d-485e-82b4-91b1e19016b7-kube-api-access-4c5wb" pod "a526e313-2c7d-485e-82b4-91b1e19016b7" (UID: "a526e313-2c7d-485e-82b4-91b1e19016b7"). InnerVolumeSpecName "kube-api-access-4c5wb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:02:56.491989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.489815 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa52ad9-568f-438f-8f40-4748c2444159-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:56.491989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.489852 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a526e313-2c7d-485e-82b4-91b1e19016b7-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:56.491989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.489867 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frcw8\" (UniqueName: \"kubernetes.io/projected/cfa52ad9-568f-438f-8f40-4748c2444159-kube-api-access-frcw8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:56.491989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.489900 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c5wb\" (UniqueName: \"kubernetes.io/projected/a526e313-2c7d-485e-82b4-91b1e19016b7-kube-api-access-4c5wb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:02:56.996980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.996948 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b"] Nov 01 16:02:56.997650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997625 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9d2067eb-c85b-4c6b-866f-339bcd750044" containerName="revoke-keys" Nov 01 16:02:56.997650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997642 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d2067eb-c85b-4c6b-866f-339bcd750044" containerName="revoke-keys" Nov 01 16:02:56.997815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997679 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a526e313-2c7d-485e-82b4-91b1e19016b7" containerName="revoke-keys" Nov 01 16:02:56.997815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997684 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a526e313-2c7d-485e-82b4-91b1e19016b7" containerName="revoke-keys" Nov 01 16:02:56.997815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997702 4573 cpu_manager.go:414] "containerMap: removing container" podUID="d797d9a4-54a5-479b-a186-8d8d21b99f58" containerName="revoke-keys" Nov 01 16:02:56.997815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997708 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d797d9a4-54a5-479b-a186-8d8d21b99f58" containerName="revoke-keys" Nov 01 16:02:56.997815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997720 4573 cpu_manager.go:414] "containerMap: removing container" podUID="cfa52ad9-568f-438f-8f40-4748c2444159" containerName="revoke-keys" Nov 01 16:02:56.997815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997726 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa52ad9-568f-438f-8f40-4748c2444159" containerName="revoke-keys" Nov 01 16:02:56.998010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997848 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d2067eb-c85b-4c6b-866f-339bcd750044" containerName="revoke-keys" Nov 01 16:02:56.998010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997864 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a526e313-2c7d-485e-82b4-91b1e19016b7" containerName="revoke-keys" Nov 01 16:02:56.998010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997876 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="d797d9a4-54a5-479b-a186-8d8d21b99f58" containerName="revoke-keys" Nov 01 16:02:56.998010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:56.997887 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa52ad9-568f-438f-8f40-4748c2444159" containerName="revoke-keys" Nov 01 16:02:57.009084 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.007419 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.017014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.011774 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b"] Nov 01 16:02:57.018167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.017434 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" event={"ID":"a526e313-2c7d-485e-82b4-91b1e19016b7","Type":"ContainerDied","Data":"7f000eca75905c7e8f8e75592bf7b66bcdfd13853caf4abeb7905b46dba34977"} Nov 01 16:02:57.018167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.017514 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f000eca75905c7e8f8e75592bf7b66bcdfd13853caf4abeb7905b46dba34977" Nov 01 16:02:57.018167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.017655 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78" Nov 01 16:02:57.022641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.020101 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" event={"ID":"cfa52ad9-568f-438f-8f40-4748c2444159","Type":"ContainerDied","Data":"3b2a1cb4bb363018a9c029aa5d6b2ec856675319bbb0b02681aaa68ed691c29c"} Nov 01 16:02:57.022641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.020172 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b2a1cb4bb363018a9c029aa5d6b2ec856675319bbb0b02681aaa68ed691c29c" Nov 01 16:02:57.022641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.020295 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk" Nov 01 16:02:57.098410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.098377 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.098573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.098523 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2nzd\" (UniqueName: \"kubernetes.io/projected/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-kube-api-access-m2nzd\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.098865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.098846 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.131062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.131032 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:57.210530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.201361 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.210530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.201476 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.210530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.201576 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2nzd\" (UniqueName: \"kubernetes.io/projected/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-kube-api-access-m2nzd\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.210530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.202560 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.210530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.205343 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.220209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.216530 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2nzd\" (UniqueName: \"kubernetes.io/projected/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-kube-api-access-m2nzd\") pod \"maas-api-revoke-sub-690413d9-k2g2b\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.340405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.340330 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:02:57.627382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.627347 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b"] Nov 01 16:02:57.627533 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:02:57.627361 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fcbe13_b3b0_4c33_8a7e_e97c2aff7ac5.slice/crio-f81f40701556b1550ecd42157792ae62fe286825062064033c83897f894b4c8d WatchSource:0}: Error finding container f81f40701556b1550ecd42157792ae62fe286825062064033c83897f894b4c8d: Status 404 returned error can't find the container with id f81f40701556b1550ecd42157792ae62fe286825062064033c83897f894b4c8d Nov 01 16:02:57.918218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:57.918142 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:58.054169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:58.052532 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" event={"ID":"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5","Type":"ContainerStarted","Data":"3acf874dfa957d7118e3e130058444f8ccebd8ad8e13f6fa9e75b7bd6cdbf304"} Nov 01 16:02:58.054169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:58.052594 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" event={"ID":"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5","Type":"ContainerStarted","Data":"f81f40701556b1550ecd42157792ae62fe286825062064033c83897f894b4c8d"} Nov 01 16:02:58.077194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:58.077130 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" podStartSLOduration=2.077110515 podStartE2EDuration="2.077110515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:02:58.068917097 +0000 UTC m=+915.772083034" watchObservedRunningTime="2026-11-01 16:02:58.077110515 +0000 UTC m=+915.780276451" Nov 01 16:02:58.677397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:58.677311 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:02:58.759500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:58.759460 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:02:59.064817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:59.064696 4573 generic.go:356] "Generic (PLEG): container finished" podID="91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5" containerID="3acf874dfa957d7118e3e130058444f8ccebd8ad8e13f6fa9e75b7bd6cdbf304" exitCode=0 Nov 01 16:02:59.064817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:59.064786 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" event={"ID":"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5","Type":"ContainerDied","Data":"3acf874dfa957d7118e3e130058444f8ccebd8ad8e13f6fa9e75b7bd6cdbf304"} Nov 01 16:02:59.476130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:02:59.476093 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:01.046777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.024289 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:03:01.417004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.379280 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:01.462847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.434294 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.462847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.435638 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:01.567253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.567202 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.691717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.690876 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:01.786041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.786008 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:03:01.952402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.952301 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-kube-api-access-m2nzd\" (UniqueName: \"kubernetes.io/projected/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-kube-api-access-m2nzd\") pod \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " Nov 01 16:03:01.967130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:01.967089 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-xks-ca\") pod \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " Nov 01 16:03:02.002679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.002635 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-openshift-ca\") pod \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\" (UID: \"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5\") " Nov 01 16:03:02.003909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.003879 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-openshift-ca" pod "91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5" (UID: "91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:02.028776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.008113 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-kube-api-access-m2nzd" pod "91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5" (UID: "91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5"). InnerVolumeSpecName "kube-api-access-m2nzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:02.029527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.029159 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.029889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.029864 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:03:02.093774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.090980 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:03:02.094531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.094498 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-xks-ca" pod "91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5" (UID: "91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:02.146776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.113282 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:02.146776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.113318 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:02.146776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.113332 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2nzd\" (UniqueName: \"kubernetes.io/projected/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5-kube-api-access-m2nzd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:02.375780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.373607 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:02.375780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.374032 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:02.453776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.440113 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.453776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.449356 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:03:02.453776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.449787 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:03:02.453776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.449845 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:03:02.483327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.457814 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 01 16:03:02.483327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.458136 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:02.542774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.510031 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:02.568779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.543739 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:03:02.725601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.725499 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.761156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.726101 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.804646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.804590 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:02.930805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:02.913329 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" Nov 01 16:03:03.040833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.035875 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:03:03.098777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.095849 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.135791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.107136 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:03:03.135791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.107275 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b" event={"ID":"91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5","Type":"ContainerDied","Data":"f81f40701556b1550ecd42157792ae62fe286825062064033c83897f894b4c8d"} Nov 01 16:03:03.135791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.107307 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f81f40701556b1550ecd42157792ae62fe286825062064033c83897f894b4c8d" Nov 01 16:03:03.135791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.107681 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:03:03.135791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.107738 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:03:03.384915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.369725 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:03.384915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.369814 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.411771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.411015 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:03.411771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.411292 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:03.412026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.411968 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:03:03.432878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.412028 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:03:03.458059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.445054 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.458059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.451834 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.462744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.462399 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:03.588776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.552086 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:03.770112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.770008 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:03:03.774983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:03.774931 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.113320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.112007 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:03:04.299516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.299460 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:03:04.484904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.484851 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:04.733818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.485194 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.733818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.485262 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:04.733818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.485283 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:04.788831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.515880 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.788831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.682343 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Nov 01 16:03:04.788831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.785076 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:04.842777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:04.803897 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.178499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.163716 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:03:05.327040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.305056 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.757777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.729647 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.757777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.730069 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:05.757777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.730364 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:05.757777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.730618 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:05.823822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.796376 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.823822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.797230 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:03:05.823822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:05.805293 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:05.926798 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:05.917343 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.418s" Nov 01 16:03:06.028057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.023847 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.189144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.189101 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:03:06.308799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.307952 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:06.437782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.428823 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj"] Nov 01 16:03:06.437782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.430005 4573 cpu_manager.go:414] "containerMap: removing container" podUID="91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5" containerName="revoke-keys" Nov 01 16:03:06.437782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.430031 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5" containerName="revoke-keys" Nov 01 16:03:06.437782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.430233 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5" containerName="revoke-keys" Nov 01 16:03:06.442400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.442375 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:06.453839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.452478 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:03:06.510792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.504888 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:03:06.599304 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.599198 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:06.599647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.599610 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:06.599883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.599866 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmvkb\" (UniqueName: \"kubernetes.io/projected/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-kube-api-access-mmvkb\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:06.625785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.618624 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:03:06.625785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.618682 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj"] Nov 01 16:03:06.784074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.782471 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:03:06.796157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.785420 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:06.796157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.785852 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:06.796157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.786128 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:06.796157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.786375 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:06.935778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.917893 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:06.935778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.918000 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:06.935778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.918089 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmvkb\" (UniqueName: \"kubernetes.io/projected/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-kube-api-access-mmvkb\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:06.951658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:06.950815 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:07.027551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.027104 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:03:07.040796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.035827 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:07.051481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.051449 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:03:07.061054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.051993 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:03:07.139164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.061946 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:03:07.155046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.073145 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmvkb\" (UniqueName: \"kubernetes.io/projected/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-kube-api-access-mmvkb\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:07.155291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.155172 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:03:07.155489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.155461 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:03:07.162056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.161986 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:03:07.164200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.164183 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:03:07.168569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.168549 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:03:07.169632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.169610 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264926e3-rg6jj\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:07.432405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.432373 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:07.557934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.557885 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:07.740571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:07.738970 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj"] Nov 01 16:03:08.171914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.171874 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" event={"ID":"bc9e4b7d-3fd8-4883-be91-e57d10e5e957","Type":"ContainerStarted","Data":"0445caa5246c9ce5330c33c57e3887fff31655758990a087a6c60d9e426a9114"} Nov 01 16:03:08.172081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.171934 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" event={"ID":"bc9e4b7d-3fd8-4883-be91-e57d10e5e957","Type":"ContainerStarted","Data":"47c608f20c764d7c0b2fe76ef07143259ec92464f5ff797be5b58c9bce257bda"} Nov 01 16:03:08.186914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.186886 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8"] Nov 01 16:03:08.188769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.188711 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" podStartSLOduration=3.188693976 podStartE2EDuration="3.188693976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:08.188014484 +0000 UTC m=+925.891180431" watchObservedRunningTime="2026-11-01 16:03:08.188693976 +0000 UTC m=+925.891859913" Nov 01 16:03:08.192028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.192002 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.201434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.201404 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8"] Nov 01 16:03:08.284900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.284815 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/3c949b85-1f70-447c-8016-c3f1d6d25714-kube-api-access-b5htb\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.285124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.285095 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.285205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.285143 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.362472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.362440 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:08.386395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.386371 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.386536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.386420 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.386616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.386554 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/3c949b85-1f70-447c-8016-c3f1d6d25714-kube-api-access-b5htb\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.387058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.387037 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.395292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.395255 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.400780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.400697 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/3c949b85-1f70-447c-8016-c3f1d6d25714-kube-api-access-b5htb\") pod \"maas-api-revoke-sub-beb88848-2gvm8\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.450079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.450048 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66"] Nov 01 16:03:08.459777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.455875 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.474930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.474896 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66"] Nov 01 16:03:08.487690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.487658 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.487855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.487833 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzrxc\" (UniqueName: \"kubernetes.io/projected/afa44df9-6058-416f-a610-bf5ec65c41b5-kube-api-access-fzrxc\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.487913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.487870 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.512380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.512353 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:08.596968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.596928 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.597159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.597095 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzrxc\" (UniqueName: \"kubernetes.io/projected/afa44df9-6058-416f-a610-bf5ec65c41b5-kube-api-access-fzrxc\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.597159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.597135 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.597660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.597637 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.599936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.599891 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.606248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.606222 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzrxc\" (UniqueName: \"kubernetes.io/projected/afa44df9-6058-416f-a610-bf5ec65c41b5-kube-api-access-fzrxc\") pod \"maas-api-revoke-sub-b2412c0c-xst66\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.785068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.785038 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:08.801307 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:08.801279 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c949b85_1f70_447c_8016_c3f1d6d25714.slice/crio-f9126ecce4e95fff8dbb250367f3ed221fefaf1b673372f53a37e99e89fdc6f7 WatchSource:0}: Error finding container f9126ecce4e95fff8dbb250367f3ed221fefaf1b673372f53a37e99e89fdc6f7: Status 404 returned error can't find the container with id f9126ecce4e95fff8dbb250367f3ed221fefaf1b673372f53a37e99e89fdc6f7 Nov 01 16:03:08.803963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:08.803941 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8"] Nov 01 16:03:09.073206 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:09.073147 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa44df9_6058_416f_a610_bf5ec65c41b5.slice/crio-eea0b5b584ed6c3bd8f350bc113ca002b2fcb99af37fbd42341c501cae6d7849 WatchSource:0}: Error finding container eea0b5b584ed6c3bd8f350bc113ca002b2fcb99af37fbd42341c501cae6d7849: Status 404 returned error can't find the container with id eea0b5b584ed6c3bd8f350bc113ca002b2fcb99af37fbd42341c501cae6d7849 Nov 01 16:03:09.074765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.074729 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66"] Nov 01 16:03:09.173917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.173883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:09.181708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.181681 4573 generic.go:356] "Generic (PLEG): container finished" podID="bc9e4b7d-3fd8-4883-be91-e57d10e5e957" containerID="0445caa5246c9ce5330c33c57e3887fff31655758990a087a6c60d9e426a9114" exitCode=0 Nov 01 16:03:09.181889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.181776 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" event={"ID":"bc9e4b7d-3fd8-4883-be91-e57d10e5e957","Type":"ContainerDied","Data":"0445caa5246c9ce5330c33c57e3887fff31655758990a087a6c60d9e426a9114"} Nov 01 16:03:09.183141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.183117 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" event={"ID":"afa44df9-6058-416f-a610-bf5ec65c41b5","Type":"ContainerStarted","Data":"eea0b5b584ed6c3bd8f350bc113ca002b2fcb99af37fbd42341c501cae6d7849"} Nov 01 16:03:09.184698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.184673 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" event={"ID":"3c949b85-1f70-447c-8016-c3f1d6d25714","Type":"ContainerStarted","Data":"ecf1a6a991e0be2f993d97225444f2868f73b3c2b106aa0e96539be4fc30290e"} Nov 01 16:03:09.184803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.184715 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" event={"ID":"3c949b85-1f70-447c-8016-c3f1d6d25714","Type":"ContainerStarted","Data":"f9126ecce4e95fff8dbb250367f3ed221fefaf1b673372f53a37e99e89fdc6f7"} Nov 01 16:03:09.240881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.240818 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" podStartSLOduration=1.240740889 podStartE2EDuration="1.240740889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:09.240204528 +0000 UTC m=+926.943370465" watchObservedRunningTime="2026-11-01 16:03:09.240740889 +0000 UTC m=+926.943906825" Nov 01 16:03:09.360129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.360102 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9"] Nov 01 16:03:09.366233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.366200 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.376806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.374714 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9"] Nov 01 16:03:09.410210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.410173 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006667a3-a418-4fea-abd2-6254d841716d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.410384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.410332 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006667a3-a418-4fea-abd2-6254d841716d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.410493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.410467 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5n8j\" (UniqueName: \"kubernetes.io/projected/006667a3-a418-4fea-abd2-6254d841716d-kube-api-access-t5n8j\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.512141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.512102 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5n8j\" (UniqueName: \"kubernetes.io/projected/006667a3-a418-4fea-abd2-6254d841716d-kube-api-access-t5n8j\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.512628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.512237 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006667a3-a418-4fea-abd2-6254d841716d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.512628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.512430 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006667a3-a418-4fea-abd2-6254d841716d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.513089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.513066 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006667a3-a418-4fea-abd2-6254d841716d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.514636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.514613 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006667a3-a418-4fea-abd2-6254d841716d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.522266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.522245 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5n8j\" (UniqueName: \"kubernetes.io/projected/006667a3-a418-4fea-abd2-6254d841716d-kube-api-access-t5n8j\") pod \"maas-api-revoke-sub-758192c0-cxkg9\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.686324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.686292 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:09.969481 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:09.969453 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006667a3_a418_4fea_abd2_6254d841716d.slice/crio-47d1e5f60f2e4e68f7ba8a8995fbda075bbf77601a8489363ec430d994d2eac6 WatchSource:0}: Error finding container 47d1e5f60f2e4e68f7ba8a8995fbda075bbf77601a8489363ec430d994d2eac6: Status 404 returned error can't find the container with id 47d1e5f60f2e4e68f7ba8a8995fbda075bbf77601a8489363ec430d994d2eac6 Nov 01 16:03:09.970976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:09.970950 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9"] Nov 01 16:03:10.042852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.040835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:10.198004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.197965 4573 generic.go:356] "Generic (PLEG): container finished" podID="3c949b85-1f70-447c-8016-c3f1d6d25714" containerID="ecf1a6a991e0be2f993d97225444f2868f73b3c2b106aa0e96539be4fc30290e" exitCode=0 Nov 01 16:03:10.198172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.198004 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" event={"ID":"3c949b85-1f70-447c-8016-c3f1d6d25714","Type":"ContainerDied","Data":"ecf1a6a991e0be2f993d97225444f2868f73b3c2b106aa0e96539be4fc30290e"} Nov 01 16:03:10.199418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.199396 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" event={"ID":"006667a3-a418-4fea-abd2-6254d841716d","Type":"ContainerStarted","Data":"47d1e5f60f2e4e68f7ba8a8995fbda075bbf77601a8489363ec430d994d2eac6"} Nov 01 16:03:10.201041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.201019 4573 generic.go:356] "Generic (PLEG): container finished" podID="afa44df9-6058-416f-a610-bf5ec65c41b5" containerID="ee267ad91273fb140aaeeff62dd7a27befd2238e9be51c356be3f2a8096eb523" exitCode=0 Nov 01 16:03:10.201120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.201046 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" event={"ID":"afa44df9-6058-416f-a610-bf5ec65c41b5","Type":"ContainerDied","Data":"ee267ad91273fb140aaeeff62dd7a27befd2238e9be51c356be3f2a8096eb523"} Nov 01 16:03:10.382149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.382122 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:10.442020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.441991 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-kube-api-access-mmvkb\" (UniqueName: \"kubernetes.io/projected/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-kube-api-access-mmvkb\") pod \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " Nov 01 16:03:10.442208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.442071 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-xks-ca\") pod \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " Nov 01 16:03:10.442208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.442149 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-openshift-ca\") pod \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\" (UID: \"bc9e4b7d-3fd8-4883-be91-e57d10e5e957\") " Nov 01 16:03:10.442564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.442537 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-openshift-ca" pod "bc9e4b7d-3fd8-4883-be91-e57d10e5e957" (UID: "bc9e4b7d-3fd8-4883-be91-e57d10e5e957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:10.443013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.442993 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:10.444707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.444676 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-xks-ca" pod "bc9e4b7d-3fd8-4883-be91-e57d10e5e957" (UID: "bc9e4b7d-3fd8-4883-be91-e57d10e5e957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:10.444934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.444912 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-kube-api-access-mmvkb" pod "bc9e4b7d-3fd8-4883-be91-e57d10e5e957" (UID: "bc9e4b7d-3fd8-4883-be91-e57d10e5e957"). InnerVolumeSpecName "kube-api-access-mmvkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:10.544266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.544193 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmvkb\" (UniqueName: \"kubernetes.io/projected/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-kube-api-access-mmvkb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:10.544266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.544220 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc9e4b7d-3fd8-4883-be91-e57d10e5e957-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:10.784927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:10.784890 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:11.043244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.043200 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd"] Nov 01 16:03:11.044255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.044229 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bc9e4b7d-3fd8-4883-be91-e57d10e5e957" containerName="revoke-keys" Nov 01 16:03:11.044255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.044254 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9e4b7d-3fd8-4883-be91-e57d10e5e957" containerName="revoke-keys" Nov 01 16:03:11.044398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.044368 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9e4b7d-3fd8-4883-be91-e57d10e5e957" containerName="revoke-keys" Nov 01 16:03:11.052151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.052112 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.057203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.057173 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd"] Nov 01 16:03:11.195883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.195843 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xstxr\" (UniqueName: \"kubernetes.io/projected/fabc4f98-e053-40ce-9b8f-46c988a68604-kube-api-access-xstxr\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.196036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.196015 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.196239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.196223 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.211409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.211379 4573 generic.go:356] "Generic (PLEG): container finished" podID="006667a3-a418-4fea-abd2-6254d841716d" containerID="8f680718532f5f1f1a5a1d9f45da50144cf8f828bb8886364b3ffa066e7867d5" exitCode=0 Nov 01 16:03:11.211532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.211463 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" event={"ID":"006667a3-a418-4fea-abd2-6254d841716d","Type":"ContainerDied","Data":"8f680718532f5f1f1a5a1d9f45da50144cf8f828bb8886364b3ffa066e7867d5"} Nov 01 16:03:11.212911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.212893 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" Nov 01 16:03:11.213028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.212919 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj" event={"ID":"bc9e4b7d-3fd8-4883-be91-e57d10e5e957","Type":"ContainerDied","Data":"47c608f20c764d7c0b2fe76ef07143259ec92464f5ff797be5b58c9bce257bda"} Nov 01 16:03:11.213028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.212958 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47c608f20c764d7c0b2fe76ef07143259ec92464f5ff797be5b58c9bce257bda" Nov 01 16:03:11.298380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.298294 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.298681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.298576 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.298779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.298706 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xstxr\" (UniqueName: \"kubernetes.io/projected/fabc4f98-e053-40ce-9b8f-46c988a68604-kube-api-access-xstxr\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.299133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.299106 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.308116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.308095 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.310523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.310484 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xstxr\" (UniqueName: \"kubernetes.io/projected/fabc4f98-e053-40ce-9b8f-46c988a68604-kube-api-access-xstxr\") pod \"maas-api-revoke-sub-a86d2fb4-bnxxd\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.384870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.384832 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:11.429123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.429082 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:11.438557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.438436 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:11.579327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.579236 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:11.602612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.602584 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-xks-ca\") pod \"3c949b85-1f70-447c-8016-c3f1d6d25714\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " Nov 01 16:03:11.602796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.602636 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa44df9-6058-416f-a610-bf5ec65c41b5-kube-api-access-fzrxc\" (UniqueName: \"kubernetes.io/projected/afa44df9-6058-416f-a610-bf5ec65c41b5-kube-api-access-fzrxc\") pod \"afa44df9-6058-416f-a610-bf5ec65c41b5\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " Nov 01 16:03:11.602796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.602673 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-openshift-ca\") pod \"3c949b85-1f70-447c-8016-c3f1d6d25714\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " Nov 01 16:03:11.602796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.602738 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c949b85-1f70-447c-8016-c3f1d6d25714-kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/3c949b85-1f70-447c-8016-c3f1d6d25714-kube-api-access-b5htb\") pod \"3c949b85-1f70-447c-8016-c3f1d6d25714\" (UID: \"3c949b85-1f70-447c-8016-c3f1d6d25714\") " Nov 01 16:03:11.602990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.602900 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-openshift-ca\") pod \"afa44df9-6058-416f-a610-bf5ec65c41b5\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " Nov 01 16:03:11.603119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.603100 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-xks-ca\") pod \"afa44df9-6058-416f-a610-bf5ec65c41b5\" (UID: \"afa44df9-6058-416f-a610-bf5ec65c41b5\") " Nov 01 16:03:11.603547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.603229 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-openshift-ca" pod "3c949b85-1f70-447c-8016-c3f1d6d25714" (UID: "3c949b85-1f70-447c-8016-c3f1d6d25714"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:11.603547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.603485 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-openshift-ca" pod "afa44df9-6058-416f-a610-bf5ec65c41b5" (UID: "afa44df9-6058-416f-a610-bf5ec65c41b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:11.604091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.604066 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:11.604212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.604198 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:11.605465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.605429 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa44df9-6058-416f-a610-bf5ec65c41b5-kube-api-access-fzrxc" pod "afa44df9-6058-416f-a610-bf5ec65c41b5" (UID: "afa44df9-6058-416f-a610-bf5ec65c41b5"). InnerVolumeSpecName "kube-api-access-fzrxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:11.605838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.605816 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-xks-ca" pod "3c949b85-1f70-447c-8016-c3f1d6d25714" (UID: "3c949b85-1f70-447c-8016-c3f1d6d25714"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:11.606080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.606055 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c949b85-1f70-447c-8016-c3f1d6d25714-kube-api-access-b5htb" pod "3c949b85-1f70-447c-8016-c3f1d6d25714" (UID: "3c949b85-1f70-447c-8016-c3f1d6d25714"). InnerVolumeSpecName "kube-api-access-b5htb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:11.606295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.606269 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-xks-ca" pod "afa44df9-6058-416f-a610-bf5ec65c41b5" (UID: "afa44df9-6058-416f-a610-bf5ec65c41b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:11.704653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.704622 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd"] Nov 01 16:03:11.705700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.705682 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa44df9-6058-416f-a610-bf5ec65c41b5-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:11.705914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.705703 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c949b85-1f70-447c-8016-c3f1d6d25714-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:11.705914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.705714 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzrxc\" (UniqueName: \"kubernetes.io/projected/afa44df9-6058-416f-a610-bf5ec65c41b5-kube-api-access-fzrxc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:11.705914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:11.705728 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5htb\" (UniqueName: \"kubernetes.io/projected/3c949b85-1f70-447c-8016-c3f1d6d25714-kube-api-access-b5htb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:12.223285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.223203 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" Nov 01 16:03:12.223285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.223244 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8" event={"ID":"3c949b85-1f70-447c-8016-c3f1d6d25714","Type":"ContainerDied","Data":"f9126ecce4e95fff8dbb250367f3ed221fefaf1b673372f53a37e99e89fdc6f7"} Nov 01 16:03:12.223494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.223292 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9126ecce4e95fff8dbb250367f3ed221fefaf1b673372f53a37e99e89fdc6f7" Nov 01 16:03:12.225158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.225131 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" event={"ID":"fabc4f98-e053-40ce-9b8f-46c988a68604","Type":"ContainerStarted","Data":"83a496510e4836712cbf1afd6e053ae4e01cc12557db6b7c180b3356a00fa089"} Nov 01 16:03:12.225276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.225173 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" event={"ID":"fabc4f98-e053-40ce-9b8f-46c988a68604","Type":"ContainerStarted","Data":"32ace6bdcec92b42ecebd95eeacddf0c776470ebc80eb6b0a798bab24892809d"} Nov 01 16:03:12.226698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.226680 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" Nov 01 16:03:12.226841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.226715 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66" event={"ID":"afa44df9-6058-416f-a610-bf5ec65c41b5","Type":"ContainerDied","Data":"eea0b5b584ed6c3bd8f350bc113ca002b2fcb99af37fbd42341c501cae6d7849"} Nov 01 16:03:12.226841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.226768 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eea0b5b584ed6c3bd8f350bc113ca002b2fcb99af37fbd42341c501cae6d7849" Nov 01 16:03:12.242363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.242306 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" podStartSLOduration=1.242290878 podStartE2EDuration="1.242290878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:12.240142626 +0000 UTC m=+929.943308561" watchObservedRunningTime="2026-11-01 16:03:12.242290878 +0000 UTC m=+929.945456857" Nov 01 16:03:12.362845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.360552 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws"] Nov 01 16:03:12.362845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.361581 4573 cpu_manager.go:414] "containerMap: removing container" podUID="afa44df9-6058-416f-a610-bf5ec65c41b5" containerName="revoke-keys" Nov 01 16:03:12.362845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.361598 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa44df9-6058-416f-a610-bf5ec65c41b5" containerName="revoke-keys" Nov 01 16:03:12.362845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.361658 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3c949b85-1f70-447c-8016-c3f1d6d25714" containerName="revoke-keys" Nov 01 16:03:12.362845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.361665 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c949b85-1f70-447c-8016-c3f1d6d25714" containerName="revoke-keys" Nov 01 16:03:12.362845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.361859 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c949b85-1f70-447c-8016-c3f1d6d25714" containerName="revoke-keys" Nov 01 16:03:12.362845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.361878 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa44df9-6058-416f-a610-bf5ec65c41b5" containerName="revoke-keys" Nov 01 16:03:12.388420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.388380 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws"] Nov 01 16:03:12.388586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.388511 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.389989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.389952 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:12.526527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.525035 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.526527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.525085 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctb8x\" (UniqueName: \"kubernetes.io/projected/453c59b1-3f5c-42b6-b61d-545c0a64b94c-kube-api-access-ctb8x\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.526527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.525193 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.560585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.560544 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:12.626522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.626490 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.626990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.626538 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctb8x\" (UniqueName: \"kubernetes.io/projected/453c59b1-3f5c-42b6-b61d-545c0a64b94c-kube-api-access-ctb8x\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.626990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.626635 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.627315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.627276 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.629169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.629143 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.634654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.634629 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctb8x\" (UniqueName: \"kubernetes.io/projected/453c59b1-3f5c-42b6-b61d-545c0a64b94c-kube-api-access-ctb8x\") pod \"maas-api-revoke-sub-79fb4e2d-6qkws\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.727653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.727623 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/006667a3-a418-4fea-abd2-6254d841716d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006667a3-a418-4fea-abd2-6254d841716d-maas-api-openshift-ca\") pod \"006667a3-a418-4fea-abd2-6254d841716d\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " Nov 01 16:03:12.727852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.727837 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/006667a3-a418-4fea-abd2-6254d841716d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006667a3-a418-4fea-abd2-6254d841716d-maas-api-xks-ca\") pod \"006667a3-a418-4fea-abd2-6254d841716d\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " Nov 01 16:03:12.727943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.727914 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/006667a3-a418-4fea-abd2-6254d841716d-kube-api-access-t5n8j\" (UniqueName: \"kubernetes.io/projected/006667a3-a418-4fea-abd2-6254d841716d-kube-api-access-t5n8j\") pod \"006667a3-a418-4fea-abd2-6254d841716d\" (UID: \"006667a3-a418-4fea-abd2-6254d841716d\") " Nov 01 16:03:12.728080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.728053 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/006667a3-a418-4fea-abd2-6254d841716d-maas-api-openshift-ca" pod "006667a3-a418-4fea-abd2-6254d841716d" (UID: "006667a3-a418-4fea-abd2-6254d841716d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:12.728511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.728487 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/006667a3-a418-4fea-abd2-6254d841716d-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:12.730149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.730128 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006667a3-a418-4fea-abd2-6254d841716d-kube-api-access-t5n8j" pod "006667a3-a418-4fea-abd2-6254d841716d" (UID: "006667a3-a418-4fea-abd2-6254d841716d"). InnerVolumeSpecName "kube-api-access-t5n8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:12.730381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.730352 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/006667a3-a418-4fea-abd2-6254d841716d-maas-api-xks-ca" pod "006667a3-a418-4fea-abd2-6254d841716d" (UID: "006667a3-a418-4fea-abd2-6254d841716d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:12.785813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.785742 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:12.829803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.829771 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/006667a3-a418-4fea-abd2-6254d841716d-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:12.829953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:12.829806 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5n8j\" (UniqueName: \"kubernetes.io/projected/006667a3-a418-4fea-abd2-6254d841716d-kube-api-access-t5n8j\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:13.069157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.069126 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws"] Nov 01 16:03:13.070082 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:13.070054 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453c59b1_3f5c_42b6_b61d_545c0a64b94c.slice/crio-ba7f485fce5f779f33998f6629b8ffb4adbb2f9fa9e019a24b97ecd96cbcfe79 WatchSource:0}: Error finding container ba7f485fce5f779f33998f6629b8ffb4adbb2f9fa9e019a24b97ecd96cbcfe79: Status 404 returned error can't find the container with id ba7f485fce5f779f33998f6629b8ffb4adbb2f9fa9e019a24b97ecd96cbcfe79 Nov 01 16:03:13.180505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.180472 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:13.285743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.285662 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" event={"ID":"453c59b1-3f5c-42b6-b61d-545c0a64b94c","Type":"ContainerStarted","Data":"ba7f485fce5f779f33998f6629b8ffb4adbb2f9fa9e019a24b97ecd96cbcfe79"} Nov 01 16:03:13.288256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.288231 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" Nov 01 16:03:13.289023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.288225 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9" event={"ID":"006667a3-a418-4fea-abd2-6254d841716d","Type":"ContainerDied","Data":"47d1e5f60f2e4e68f7ba8a8995fbda075bbf77601a8489363ec430d994d2eac6"} Nov 01 16:03:13.289160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.289059 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47d1e5f60f2e4e68f7ba8a8995fbda075bbf77601a8489363ec430d994d2eac6" Nov 01 16:03:13.291213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.291183 4573 generic.go:356] "Generic (PLEG): container finished" podID="fabc4f98-e053-40ce-9b8f-46c988a68604" containerID="83a496510e4836712cbf1afd6e053ae4e01cc12557db6b7c180b3356a00fa089" exitCode=0 Nov 01 16:03:13.291330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.291247 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" event={"ID":"fabc4f98-e053-40ce-9b8f-46c988a68604","Type":"ContainerDied","Data":"83a496510e4836712cbf1afd6e053ae4e01cc12557db6b7c180b3356a00fa089"} Nov 01 16:03:13.876867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:13.876825 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:14.301398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.301308 4573 generic.go:356] "Generic (PLEG): container finished" podID="453c59b1-3f5c-42b6-b61d-545c0a64b94c" containerID="286ae588a1f17eb0327bacb67457682ff6bbb6537591f2381a1736937e562d65" exitCode=0 Nov 01 16:03:14.301398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.301385 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" event={"ID":"453c59b1-3f5c-42b6-b61d-545c0a64b94c","Type":"ContainerDied","Data":"286ae588a1f17eb0327bacb67457682ff6bbb6537591f2381a1736937e562d65"} Nov 01 16:03:14.470153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.470129 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:14.573881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.573800 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-openshift-ca\") pod \"fabc4f98-e053-40ce-9b8f-46c988a68604\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " Nov 01 16:03:14.573881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.573857 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fabc4f98-e053-40ce-9b8f-46c988a68604-kube-api-access-xstxr\" (UniqueName: \"kubernetes.io/projected/fabc4f98-e053-40ce-9b8f-46c988a68604-kube-api-access-xstxr\") pod \"fabc4f98-e053-40ce-9b8f-46c988a68604\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " Nov 01 16:03:14.574107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.573883 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-xks-ca\") pod \"fabc4f98-e053-40ce-9b8f-46c988a68604\" (UID: \"fabc4f98-e053-40ce-9b8f-46c988a68604\") " Nov 01 16:03:14.574251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.574226 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-openshift-ca" pod "fabc4f98-e053-40ce-9b8f-46c988a68604" (UID: "fabc4f98-e053-40ce-9b8f-46c988a68604"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:14.574629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.574610 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:14.576029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.576000 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-xks-ca" pod "fabc4f98-e053-40ce-9b8f-46c988a68604" (UID: "fabc4f98-e053-40ce-9b8f-46c988a68604"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:14.576162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.576137 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabc4f98-e053-40ce-9b8f-46c988a68604-kube-api-access-xstxr" pod "fabc4f98-e053-40ce-9b8f-46c988a68604" (UID: "fabc4f98-e053-40ce-9b8f-46c988a68604"). InnerVolumeSpecName "kube-api-access-xstxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:14.676162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.676136 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xstxr\" (UniqueName: \"kubernetes.io/projected/fabc4f98-e053-40ce-9b8f-46c988a68604-kube-api-access-xstxr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:14.676162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.676163 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fabc4f98-e053-40ce-9b8f-46c988a68604-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:14.700784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:14.700735 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:15.321571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.321542 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" Nov 01 16:03:15.321571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.321564 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd" event={"ID":"fabc4f98-e053-40ce-9b8f-46c988a68604","Type":"ContainerDied","Data":"32ace6bdcec92b42ecebd95eeacddf0c776470ebc80eb6b0a798bab24892809d"} Nov 01 16:03:15.322063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.321644 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32ace6bdcec92b42ecebd95eeacddf0c776470ebc80eb6b0a798bab24892809d" Nov 01 16:03:15.489165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.489140 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:15.522792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.510205 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:15.702306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.702271 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453c59b1-3f5c-42b6-b61d-545c0a64b94c-kube-api-access-ctb8x\" (UniqueName: \"kubernetes.io/projected/453c59b1-3f5c-42b6-b61d-545c0a64b94c-kube-api-access-ctb8x\") pod \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " Nov 01 16:03:15.702487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.702400 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-openshift-ca\") pod \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " Nov 01 16:03:15.702487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.702430 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-xks-ca\") pod \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\" (UID: \"453c59b1-3f5c-42b6-b61d-545c0a64b94c\") " Nov 01 16:03:15.702742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.702711 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-openshift-ca" pod "453c59b1-3f5c-42b6-b61d-545c0a64b94c" (UID: "453c59b1-3f5c-42b6-b61d-545c0a64b94c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:15.703118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.703097 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:15.704556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.704533 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-xks-ca" pod "453c59b1-3f5c-42b6-b61d-545c0a64b94c" (UID: "453c59b1-3f5c-42b6-b61d-545c0a64b94c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:15.704633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.704594 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453c59b1-3f5c-42b6-b61d-545c0a64b94c-kube-api-access-ctb8x" pod "453c59b1-3f5c-42b6-b61d-545c0a64b94c" (UID: "453c59b1-3f5c-42b6-b61d-545c0a64b94c"). InnerVolumeSpecName "kube-api-access-ctb8x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:15.805121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.805085 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctb8x\" (UniqueName: \"kubernetes.io/projected/453c59b1-3f5c-42b6-b61d-545c0a64b94c-kube-api-access-ctb8x\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:15.805121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:15.805115 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/453c59b1-3f5c-42b6-b61d-545c0a64b94c-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:16.332275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:16.332242 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" Nov 01 16:03:16.332659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:16.332280 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws" event={"ID":"453c59b1-3f5c-42b6-b61d-545c0a64b94c","Type":"ContainerDied","Data":"ba7f485fce5f779f33998f6629b8ffb4adbb2f9fa9e019a24b97ecd96cbcfe79"} Nov 01 16:03:16.332659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:16.332326 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba7f485fce5f779f33998f6629b8ffb4adbb2f9fa9e019a24b97ecd96cbcfe79" Nov 01 16:03:16.369464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:16.369438 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:17.198388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.198354 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v"] Nov 01 16:03:17.199052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199035 4573 cpu_manager.go:414] "containerMap: removing container" podUID="006667a3-a418-4fea-abd2-6254d841716d" containerName="revoke-keys" Nov 01 16:03:17.199099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199054 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="006667a3-a418-4fea-abd2-6254d841716d" containerName="revoke-keys" Nov 01 16:03:17.199099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199079 4573 cpu_manager.go:414] "containerMap: removing container" podUID="453c59b1-3f5c-42b6-b61d-545c0a64b94c" containerName="revoke-keys" Nov 01 16:03:17.199099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199084 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453c59b1-3f5c-42b6-b61d-545c0a64b94c" containerName="revoke-keys" Nov 01 16:03:17.199253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199120 4573 cpu_manager.go:414] "containerMap: removing container" podUID="fabc4f98-e053-40ce-9b8f-46c988a68604" containerName="revoke-keys" Nov 01 16:03:17.199253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199125 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fabc4f98-e053-40ce-9b8f-46c988a68604" containerName="revoke-keys" Nov 01 16:03:17.199253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199218 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="453c59b1-3f5c-42b6-b61d-545c0a64b94c" containerName="revoke-keys" Nov 01 16:03:17.199253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199234 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="fabc4f98-e053-40ce-9b8f-46c988a68604" containerName="revoke-keys" Nov 01 16:03:17.199253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.199241 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="006667a3-a418-4fea-abd2-6254d841716d" containerName="revoke-keys" Nov 01 16:03:17.204720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.204698 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.208481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.208453 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:03:17.208613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.208588 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v"] Nov 01 16:03:17.209172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.209155 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:03:17.254270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.254235 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.254417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.254311 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcj52\" (UniqueName: \"kubernetes.io/projected/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-kube-api-access-pcj52\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.254562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.254538 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.335721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.335686 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:17.355558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.355531 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.355699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.355605 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcj52\" (UniqueName: \"kubernetes.io/projected/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-kube-api-access-pcj52\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.355789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.355772 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.356593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.356566 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.358238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.358206 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.363619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.363595 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcj52\" (UniqueName: \"kubernetes.io/projected/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-kube-api-access-pcj52\") pod \"maas-api-revoke-sub-dba73893-7lf9v\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.524272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.524175 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:17.839648 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:17.839619 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ea44c00_84b6_4cd8_9830_b20f1f5e4445.slice/crio-640089079de3e596329cd9949171c597c52be93cdf308aea19f670fcbfd2e52f WatchSource:0}: Error finding container 640089079de3e596329cd9949171c597c52be93cdf308aea19f670fcbfd2e52f: Status 404 returned error can't find the container with id 640089079de3e596329cd9949171c597c52be93cdf308aea19f670fcbfd2e52f Nov 01 16:03:17.840325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:17.840301 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v"] Nov 01 16:03:18.351394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.351355 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" event={"ID":"1ea44c00-84b6-4cd8-9830-b20f1f5e4445","Type":"ContainerStarted","Data":"c718751c833eb30717c1c20977852121227abeb027add8a1462f8cdc4e6de055"} Nov 01 16:03:18.351877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.351451 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" event={"ID":"1ea44c00-84b6-4cd8-9830-b20f1f5e4445","Type":"ContainerStarted","Data":"640089079de3e596329cd9949171c597c52be93cdf308aea19f670fcbfd2e52f"} Nov 01 16:03:18.368634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.368589 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" podStartSLOduration=1.368576053 podStartE2EDuration="1.368576053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:18.365889052 +0000 UTC m=+936.069055012" watchObservedRunningTime="2026-11-01 16:03:18.368576053 +0000 UTC m=+936.071741989" Nov 01 16:03:18.435526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.435497 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:18.606395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.606345 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl"] Nov 01 16:03:18.611299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.611273 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.619716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.619687 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl"] Nov 01 16:03:18.645672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.645639 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr"] Nov 01 16:03:18.650348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.650328 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.667946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.666653 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr"] Nov 01 16:03:18.680810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.680782 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.681111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.681087 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.681173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.681126 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcjnj\" (UniqueName: \"kubernetes.io/projected/5671ea17-8319-4db9-aff9-6854f88fb606-kube-api-access-lcjnj\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.681293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.681271 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.681337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.681313 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztclp\" (UniqueName: \"kubernetes.io/projected/d501b45a-9ba5-4db3-874f-bef00cf5b50b-kube-api-access-ztclp\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.681422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.681406 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.782736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.782702 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.782938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.782744 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztclp\" (UniqueName: \"kubernetes.io/projected/d501b45a-9ba5-4db3-874f-bef00cf5b50b-kube-api-access-ztclp\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.782938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.782807 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.783056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.782962 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.783180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.783161 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.783247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.783193 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcjnj\" (UniqueName: \"kubernetes.io/projected/5671ea17-8319-4db9-aff9-6854f88fb606-kube-api-access-lcjnj\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.783676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.783564 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.783997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.783975 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.786031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.786008 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.786153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.786134 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.790686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.790668 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcjnj\" (UniqueName: \"kubernetes.io/projected/5671ea17-8319-4db9-aff9-6854f88fb606-kube-api-access-lcjnj\") pod \"maas-api-revoke-sub-c4b8258c-rt9tl\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.790911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.790889 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztclp\" (UniqueName: \"kubernetes.io/projected/d501b45a-9ba5-4db3-874f-bef00cf5b50b-kube-api-access-ztclp\") pod \"maas-api-revoke-sub-57077ffd-xstbr\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:18.932416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.932386 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:18.974991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:18.974960 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:19.220994 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:19.220966 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5671ea17_8319_4db9_aff9_6854f88fb606.slice/crio-1f65ce38e734a99d8ce5cbc6b54ec06981eeef1b72f56d1087de4a4e107ea502 WatchSource:0}: Error finding container 1f65ce38e734a99d8ce5cbc6b54ec06981eeef1b72f56d1087de4a4e107ea502: Status 404 returned error can't find the container with id 1f65ce38e734a99d8ce5cbc6b54ec06981eeef1b72f56d1087de4a4e107ea502 Nov 01 16:03:19.223338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:19.223311 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl"] Nov 01 16:03:19.283461 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:19.283431 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd501b45a_9ba5_4db3_874f_bef00cf5b50b.slice/crio-f867c474cafd86f1f883d3d652e5fce35765659a628bc0f9c0ff005291cbe825 WatchSource:0}: Error finding container f867c474cafd86f1f883d3d652e5fce35765659a628bc0f9c0ff005291cbe825: Status 404 returned error can't find the container with id f867c474cafd86f1f883d3d652e5fce35765659a628bc0f9c0ff005291cbe825 Nov 01 16:03:19.285312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:19.285288 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr"] Nov 01 16:03:19.364309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:19.364265 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:19.370244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:19.370215 4573 generic.go:356] "Generic (PLEG): container finished" podID="1ea44c00-84b6-4cd8-9830-b20f1f5e4445" containerID="c718751c833eb30717c1c20977852121227abeb027add8a1462f8cdc4e6de055" exitCode=0 Nov 01 16:03:19.370373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:19.370292 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" event={"ID":"1ea44c00-84b6-4cd8-9830-b20f1f5e4445","Type":"ContainerDied","Data":"c718751c833eb30717c1c20977852121227abeb027add8a1462f8cdc4e6de055"} Nov 01 16:03:19.371902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:19.371876 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" event={"ID":"5671ea17-8319-4db9-aff9-6854f88fb606","Type":"ContainerStarted","Data":"1f65ce38e734a99d8ce5cbc6b54ec06981eeef1b72f56d1087de4a4e107ea502"} Nov 01 16:03:19.373272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:19.373249 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" event={"ID":"d501b45a-9ba5-4db3-874f-bef00cf5b50b","Type":"ContainerStarted","Data":"f867c474cafd86f1f883d3d652e5fce35765659a628bc0f9c0ff005291cbe825"} Nov 01 16:03:20.318170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.318142 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:20.419511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.419141 4573 generic.go:356] "Generic (PLEG): container finished" podID="5671ea17-8319-4db9-aff9-6854f88fb606" containerID="127b4c15e8ce5f7b7cfd8bb8b2adc921430554562b884be33faf1dfc51a2460d" exitCode=0 Nov 01 16:03:20.419511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.419378 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" event={"ID":"5671ea17-8319-4db9-aff9-6854f88fb606","Type":"ContainerDied","Data":"127b4c15e8ce5f7b7cfd8bb8b2adc921430554562b884be33faf1dfc51a2460d"} Nov 01 16:03:20.422922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.422277 4573 generic.go:356] "Generic (PLEG): container finished" podID="d501b45a-9ba5-4db3-874f-bef00cf5b50b" containerID="1ccafdca17c2c8aa53072000aacbfdf35a408165edcf0cf13514bc566dcf0e38" exitCode=0 Nov 01 16:03:20.422922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.422858 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" event={"ID":"d501b45a-9ba5-4db3-874f-bef00cf5b50b","Type":"ContainerDied","Data":"1ccafdca17c2c8aa53072000aacbfdf35a408165edcf0cf13514bc566dcf0e38"} Nov 01 16:03:20.619128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.619099 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:20.732648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.732569 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-kube-api-access-pcj52\" (UniqueName: \"kubernetes.io/projected/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-kube-api-access-pcj52\") pod \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " Nov 01 16:03:20.732648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.732602 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-openshift-ca\") pod \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " Nov 01 16:03:20.732841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.732814 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-xks-ca\") pod \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\" (UID: \"1ea44c00-84b6-4cd8-9830-b20f1f5e4445\") " Nov 01 16:03:20.733060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.733034 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-openshift-ca" pod "1ea44c00-84b6-4cd8-9830-b20f1f5e4445" (UID: "1ea44c00-84b6-4cd8-9830-b20f1f5e4445"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:20.733511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.733489 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:20.734809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.734786 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-xks-ca" pod "1ea44c00-84b6-4cd8-9830-b20f1f5e4445" (UID: "1ea44c00-84b6-4cd8-9830-b20f1f5e4445"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:20.734897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.734872 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-kube-api-access-pcj52" pod "1ea44c00-84b6-4cd8-9830-b20f1f5e4445" (UID: "1ea44c00-84b6-4cd8-9830-b20f1f5e4445"). InnerVolumeSpecName "kube-api-access-pcj52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:20.834598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.834571 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcj52\" (UniqueName: \"kubernetes.io/projected/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-kube-api-access-pcj52\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:20.834598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:20.834598 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ea44c00-84b6-4cd8-9830-b20f1f5e4445-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:21.069121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.069049 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:21.432616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.432592 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" Nov 01 16:03:21.433121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.432587 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v" event={"ID":"1ea44c00-84b6-4cd8-9830-b20f1f5e4445","Type":"ContainerDied","Data":"640089079de3e596329cd9949171c597c52be93cdf308aea19f670fcbfd2e52f"} Nov 01 16:03:21.433121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.432723 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="640089079de3e596329cd9949171c597c52be93cdf308aea19f670fcbfd2e52f" Nov 01 16:03:21.610649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.610622 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:21.620853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.620821 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk"] Nov 01 16:03:21.621521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.621501 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1ea44c00-84b6-4cd8-9830-b20f1f5e4445" containerName="revoke-keys" Nov 01 16:03:21.621521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.621519 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea44c00-84b6-4cd8-9830-b20f1f5e4445" containerName="revoke-keys" Nov 01 16:03:21.621697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.621546 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5671ea17-8319-4db9-aff9-6854f88fb606" containerName="revoke-keys" Nov 01 16:03:21.621697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.621552 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5671ea17-8319-4db9-aff9-6854f88fb606" containerName="revoke-keys" Nov 01 16:03:21.621697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.621676 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea44c00-84b6-4cd8-9830-b20f1f5e4445" containerName="revoke-keys" Nov 01 16:03:21.621697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.621687 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5671ea17-8319-4db9-aff9-6854f88fb606" containerName="revoke-keys" Nov 01 16:03:21.637435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.635700 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk"] Nov 01 16:03:21.637435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.635833 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.654594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.653580 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:21.760210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.760118 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-openshift-ca\") pod \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " Nov 01 16:03:21.760210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.760178 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-xks-ca\") pod \"5671ea17-8319-4db9-aff9-6854f88fb606\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " Nov 01 16:03:21.760439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.760240 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5671ea17-8319-4db9-aff9-6854f88fb606-kube-api-access-lcjnj\" (UniqueName: \"kubernetes.io/projected/5671ea17-8319-4db9-aff9-6854f88fb606-kube-api-access-lcjnj\") pod \"5671ea17-8319-4db9-aff9-6854f88fb606\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " Nov 01 16:03:21.760439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.760342 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d501b45a-9ba5-4db3-874f-bef00cf5b50b-kube-api-access-ztclp\" (UniqueName: \"kubernetes.io/projected/d501b45a-9ba5-4db3-874f-bef00cf5b50b-kube-api-access-ztclp\") pod \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " Nov 01 16:03:21.760566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.760450 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-xks-ca\") pod \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\" (UID: \"d501b45a-9ba5-4db3-874f-bef00cf5b50b\") " Nov 01 16:03:21.760618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.760555 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-openshift-ca" pod "d501b45a-9ba5-4db3-874f-bef00cf5b50b" (UID: "d501b45a-9ba5-4db3-874f-bef00cf5b50b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:21.760618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.760590 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-openshift-ca\") pod \"5671ea17-8319-4db9-aff9-6854f88fb606\" (UID: \"5671ea17-8319-4db9-aff9-6854f88fb606\") " Nov 01 16:03:21.761091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.761052 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-openshift-ca" pod "5671ea17-8319-4db9-aff9-6854f88fb606" (UID: "5671ea17-8319-4db9-aff9-6854f88fb606"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:21.761218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.761157 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7xd\" (UniqueName: \"kubernetes.io/projected/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-kube-api-access-pz7xd\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.761218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.761209 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.761336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.761235 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.761439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.761418 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:21.761439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.761440 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:21.762922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.762878 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5671ea17-8319-4db9-aff9-6854f88fb606-kube-api-access-lcjnj" pod "5671ea17-8319-4db9-aff9-6854f88fb606" (UID: "5671ea17-8319-4db9-aff9-6854f88fb606"). InnerVolumeSpecName "kube-api-access-lcjnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:21.763154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.763128 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-xks-ca" pod "d501b45a-9ba5-4db3-874f-bef00cf5b50b" (UID: "d501b45a-9ba5-4db3-874f-bef00cf5b50b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:21.763219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.763190 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-xks-ca" pod "5671ea17-8319-4db9-aff9-6854f88fb606" (UID: "5671ea17-8319-4db9-aff9-6854f88fb606"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:21.763995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.763976 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d501b45a-9ba5-4db3-874f-bef00cf5b50b-kube-api-access-ztclp" pod "d501b45a-9ba5-4db3-874f-bef00cf5b50b" (UID: "d501b45a-9ba5-4db3-874f-bef00cf5b50b"). InnerVolumeSpecName "kube-api-access-ztclp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:21.862362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.862332 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7xd\" (UniqueName: \"kubernetes.io/projected/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-kube-api-access-pz7xd\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.862497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.862380 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.862497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.862398 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.862562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.862550 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5671ea17-8319-4db9-aff9-6854f88fb606-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:21.862593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.862564 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcjnj\" (UniqueName: \"kubernetes.io/projected/5671ea17-8319-4db9-aff9-6854f88fb606-kube-api-access-lcjnj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:21.862593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.862574 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztclp\" (UniqueName: \"kubernetes.io/projected/d501b45a-9ba5-4db3-874f-bef00cf5b50b-kube-api-access-ztclp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:21.862593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.862583 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d501b45a-9ba5-4db3-874f-bef00cf5b50b-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:21.863039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.863015 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.864688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.864666 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.869710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.869688 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7xd\" (UniqueName: \"kubernetes.io/projected/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-kube-api-access-pz7xd\") pod \"maas-api-revoke-sub-0c809f72-nhkqk\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:21.958732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.958697 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:21.972686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:21.972663 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:22.299525 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:22.299497 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08d1fd8_46ba_4c0d_bcc3_270fe5582c81.slice/crio-37e3b69ceabf7e2f514c3e31a3670f30435b05863b5df8d8326aaf2e10f94dc7 WatchSource:0}: Error finding container 37e3b69ceabf7e2f514c3e31a3670f30435b05863b5df8d8326aaf2e10f94dc7: Status 404 returned error can't find the container with id 37e3b69ceabf7e2f514c3e31a3670f30435b05863b5df8d8326aaf2e10f94dc7 Nov 01 16:03:22.301466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.301441 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk"] Nov 01 16:03:22.447444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.447410 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" Nov 01 16:03:22.447919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.447409 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl" event={"ID":"5671ea17-8319-4db9-aff9-6854f88fb606","Type":"ContainerDied","Data":"1f65ce38e734a99d8ce5cbc6b54ec06981eeef1b72f56d1087de4a4e107ea502"} Nov 01 16:03:22.447919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.447546 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f65ce38e734a99d8ce5cbc6b54ec06981eeef1b72f56d1087de4a4e107ea502" Nov 01 16:03:22.458066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.456827 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" Nov 01 16:03:22.463666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.463632 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr" event={"ID":"d501b45a-9ba5-4db3-874f-bef00cf5b50b","Type":"ContainerDied","Data":"f867c474cafd86f1f883d3d652e5fce35765659a628bc0f9c0ff005291cbe825"} Nov 01 16:03:22.463847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.463696 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f867c474cafd86f1f883d3d652e5fce35765659a628bc0f9c0ff005291cbe825" Nov 01 16:03:22.463847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.463711 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" event={"ID":"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81","Type":"ContainerStarted","Data":"37e3b69ceabf7e2f514c3e31a3670f30435b05863b5df8d8326aaf2e10f94dc7"} Nov 01 16:03:22.559398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.559367 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8"] Nov 01 16:03:22.560158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.560141 4573 cpu_manager.go:414] "containerMap: removing container" podUID="d501b45a-9ba5-4db3-874f-bef00cf5b50b" containerName="revoke-keys" Nov 01 16:03:22.560222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.560160 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d501b45a-9ba5-4db3-874f-bef00cf5b50b" containerName="revoke-keys" Nov 01 16:03:22.560275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.560266 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="d501b45a-9ba5-4db3-874f-bef00cf5b50b" containerName="revoke-keys" Nov 01 16:03:22.566079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.566055 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.573056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.572865 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8"] Nov 01 16:03:22.676578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.676553 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.676693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.676618 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdjjp\" (UniqueName: \"kubernetes.io/projected/8bc986a7-9816-4138-b597-5569cc44f31e-kube-api-access-qdjjp\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.676789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.676741 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.778313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.778222 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.778313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.778295 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdjjp\" (UniqueName: \"kubernetes.io/projected/8bc986a7-9816-4138-b597-5569cc44f31e-kube-api-access-qdjjp\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.778518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.778392 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.779092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.779067 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.780720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.780696 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.786700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.786659 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdjjp\" (UniqueName: \"kubernetes.io/projected/8bc986a7-9816-4138-b597-5569cc44f31e-kube-api-access-qdjjp\") pod \"maas-api-revoke-sub-8b9fb427-qcvp8\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.810965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.810932 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:22.901794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.895178 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:22.922690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.922614 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664"] Nov 01 16:03:22.954794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.952891 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664"] Nov 01 16:03:22.954794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:22.953031 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.099261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.099225 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.099434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.099392 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkww\" (UniqueName: \"kubernetes.io/projected/775c8e45-c357-46f1-acc4-01adc262ff1d-kube-api-access-7fkww\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.099532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.099514 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.201090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.201048 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.201257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.201160 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkww\" (UniqueName: \"kubernetes.io/projected/775c8e45-c357-46f1-acc4-01adc262ff1d-kube-api-access-7fkww\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.201257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.201247 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.201952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.201929 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.203780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.203725 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.211712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.211687 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkww\" (UniqueName: \"kubernetes.io/projected/775c8e45-c357-46f1-acc4-01adc262ff1d-kube-api-access-7fkww\") pod \"maas-api-revoke-sub-357a04fb-8f664\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.240913 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:23.240887 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc986a7_9816_4138_b597_5569cc44f31e.slice/crio-fc53a65acbc927b269b6a334f65208b14077a5303a2746eb3db82aa53e7b9aa1 WatchSource:0}: Error finding container fc53a65acbc927b269b6a334f65208b14077a5303a2746eb3db82aa53e7b9aa1: Status 404 returned error can't find the container with id fc53a65acbc927b269b6a334f65208b14077a5303a2746eb3db82aa53e7b9aa1 Nov 01 16:03:23.242237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.242215 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8"] Nov 01 16:03:23.275776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.275743 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:23.466602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.466567 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" event={"ID":"8bc986a7-9816-4138-b597-5569cc44f31e","Type":"ContainerStarted","Data":"fc53a65acbc927b269b6a334f65208b14077a5303a2746eb3db82aa53e7b9aa1"} Nov 01 16:03:23.468574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.468551 4573 generic.go:356] "Generic (PLEG): container finished" podID="f08d1fd8-46ba-4c0d-bcc3-270fe5582c81" containerID="452f70ea7ad9b882c5d6347c6ecbbf2db84161676bf02b737c4be8c674651856" exitCode=0 Nov 01 16:03:23.468698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.468636 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" event={"ID":"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81","Type":"ContainerDied","Data":"452f70ea7ad9b882c5d6347c6ecbbf2db84161676bf02b737c4be8c674651856"} Nov 01 16:03:23.559250 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:23.559219 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775c8e45_c357_46f1_acc4_01adc262ff1d.slice/crio-10a75df4c2b21b9646c34a2c5fa776c0e3f7804b8c1d68f2ea484c921bcaec82 WatchSource:0}: Error finding container 10a75df4c2b21b9646c34a2c5fa776c0e3f7804b8c1d68f2ea484c921bcaec82: Status 404 returned error can't find the container with id 10a75df4c2b21b9646c34a2c5fa776c0e3f7804b8c1d68f2ea484c921bcaec82 Nov 01 16:03:23.561123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.561101 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664"] Nov 01 16:03:23.590001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:23.589966 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:24.097159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.097127 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f"] Nov 01 16:03:24.102135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.102107 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.110821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.110088 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f"] Nov 01 16:03:24.217644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.217611 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzmb\" (UniqueName: \"kubernetes.io/projected/3f5e8bd2-36db-4129-a074-83b8617ddfcf-kube-api-access-qfzmb\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.217831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.217675 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.217911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.217885 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.320247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.320158 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzmb\" (UniqueName: \"kubernetes.io/projected/3f5e8bd2-36db-4129-a074-83b8617ddfcf-kube-api-access-qfzmb\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.320415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.320311 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.320484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.320435 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.321167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.321142 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.322861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.322836 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.327646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.327623 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzmb\" (UniqueName: \"kubernetes.io/projected/3f5e8bd2-36db-4129-a074-83b8617ddfcf-kube-api-access-qfzmb\") pod \"maas-api-revoke-sub-2d351253-jq62f\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.358447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.358414 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:24.366041 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:24.366015 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:03:24.429629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.429322 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:24.513609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.513566 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" event={"ID":"775c8e45-c357-46f1-acc4-01adc262ff1d","Type":"ContainerStarted","Data":"5774c3a21e1c595ba0207a17fbd923296b858da2027e0635129b5e33f27f6da1"} Nov 01 16:03:24.514175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.513661 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" event={"ID":"775c8e45-c357-46f1-acc4-01adc262ff1d","Type":"ContainerStarted","Data":"10a75df4c2b21b9646c34a2c5fa776c0e3f7804b8c1d68f2ea484c921bcaec82"} Nov 01 16:03:24.515901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.515868 4573 generic.go:356] "Generic (PLEG): container finished" podID="8bc986a7-9816-4138-b597-5569cc44f31e" containerID="09b6c465d5ad0aee494eebc6bd5b31bdb3c4dfd89fc9a7a52f652337143cc892" exitCode=0 Nov 01 16:03:24.516315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.516289 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" event={"ID":"8bc986a7-9816-4138-b597-5569cc44f31e","Type":"ContainerDied","Data":"09b6c465d5ad0aee494eebc6bd5b31bdb3c4dfd89fc9a7a52f652337143cc892"} Nov 01 16:03:24.534428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.533049 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" podStartSLOduration=2.533031547 podStartE2EDuration="2.533031547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:24.529198074 +0000 UTC m=+942.232364011" watchObservedRunningTime="2026-11-01 16:03:24.533031547 +0000 UTC m=+942.236197482" Nov 01 16:03:24.751029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.751002 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:24.800506 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:24.800465 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5e8bd2_36db_4129_a074_83b8617ddfcf.slice/crio-8c61d6182884d74a9ab86272c479839d844816ac8a129837c1c9ff731b969859 WatchSource:0}: Error finding container 8c61d6182884d74a9ab86272c479839d844816ac8a129837c1c9ff731b969859: Status 404 returned error can't find the container with id 8c61d6182884d74a9ab86272c479839d844816ac8a129837c1c9ff731b969859 Nov 01 16:03:24.801012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.800991 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f"] Nov 01 16:03:24.837980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.837945 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-openshift-ca\") pod \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " Nov 01 16:03:24.838108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.838028 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-xks-ca\") pod \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " Nov 01 16:03:24.838180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.838159 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-kube-api-access-pz7xd\" (UniqueName: \"kubernetes.io/projected/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-kube-api-access-pz7xd\") pod \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\" (UID: \"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81\") " Nov 01 16:03:24.838318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.838297 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-openshift-ca" pod "f08d1fd8-46ba-4c0d-bcc3-270fe5582c81" (UID: "f08d1fd8-46ba-4c0d-bcc3-270fe5582c81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:24.838709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.838689 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:24.840286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.840265 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-xks-ca" pod "f08d1fd8-46ba-4c0d-bcc3-270fe5582c81" (UID: "f08d1fd8-46ba-4c0d-bcc3-270fe5582c81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:24.840457 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.840433 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-kube-api-access-pz7xd" pod "f08d1fd8-46ba-4c0d-bcc3-270fe5582c81" (UID: "f08d1fd8-46ba-4c0d-bcc3-270fe5582c81"). InnerVolumeSpecName "kube-api-access-pz7xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:24.940662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.940634 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:24.940662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.940662 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7xd\" (UniqueName: \"kubernetes.io/projected/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81-kube-api-access-pz7xd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:24.987248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:24.987224 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:25.526949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.526860 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" event={"ID":"3f5e8bd2-36db-4129-a074-83b8617ddfcf","Type":"ContainerStarted","Data":"ed8941ff71d896eeeae2771f9b1108954284b9571bd0f7d476aea8652ccc999f"} Nov 01 16:03:25.526949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.526923 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" event={"ID":"3f5e8bd2-36db-4129-a074-83b8617ddfcf","Type":"ContainerStarted","Data":"8c61d6182884d74a9ab86272c479839d844816ac8a129837c1c9ff731b969859"} Nov 01 16:03:25.528625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.528603 4573 generic.go:356] "Generic (PLEG): container finished" podID="775c8e45-c357-46f1-acc4-01adc262ff1d" containerID="5774c3a21e1c595ba0207a17fbd923296b858da2027e0635129b5e33f27f6da1" exitCode=0 Nov 01 16:03:25.528736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.528668 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" event={"ID":"775c8e45-c357-46f1-acc4-01adc262ff1d","Type":"ContainerDied","Data":"5774c3a21e1c595ba0207a17fbd923296b858da2027e0635129b5e33f27f6da1"} Nov 01 16:03:25.530140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.530122 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" Nov 01 16:03:25.530240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.530143 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk" event={"ID":"f08d1fd8-46ba-4c0d-bcc3-270fe5582c81","Type":"ContainerDied","Data":"37e3b69ceabf7e2f514c3e31a3670f30435b05863b5df8d8326aaf2e10f94dc7"} Nov 01 16:03:25.530240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.530181 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37e3b69ceabf7e2f514c3e31a3670f30435b05863b5df8d8326aaf2e10f94dc7" Nov 01 16:03:25.544796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.544742 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" podStartSLOduration=1.544730047 podStartE2EDuration="1.544730047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:25.54194208 +0000 UTC m=+943.245108016" watchObservedRunningTime="2026-11-01 16:03:25.544730047 +0000 UTC m=+943.247896047" Nov 01 16:03:25.630652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.630613 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:25.762691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.762654 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:25.856277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.856246 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-openshift-ca\") pod \"8bc986a7-9816-4138-b597-5569cc44f31e\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " Nov 01 16:03:25.856422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.856332 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bc986a7-9816-4138-b597-5569cc44f31e-kube-api-access-qdjjp\" (UniqueName: \"kubernetes.io/projected/8bc986a7-9816-4138-b597-5569cc44f31e-kube-api-access-qdjjp\") pod \"8bc986a7-9816-4138-b597-5569cc44f31e\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " Nov 01 16:03:25.856643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.856498 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-xks-ca\") pod \"8bc986a7-9816-4138-b597-5569cc44f31e\" (UID: \"8bc986a7-9816-4138-b597-5569cc44f31e\") " Nov 01 16:03:25.856719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.856666 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-openshift-ca" pod "8bc986a7-9816-4138-b597-5569cc44f31e" (UID: "8bc986a7-9816-4138-b597-5569cc44f31e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:25.857147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.857127 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:25.858491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.858455 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-xks-ca" pod "8bc986a7-9816-4138-b597-5569cc44f31e" (UID: "8bc986a7-9816-4138-b597-5569cc44f31e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:25.858491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.858482 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bc986a7-9816-4138-b597-5569cc44f31e-kube-api-access-qdjjp" pod "8bc986a7-9816-4138-b597-5569cc44f31e" (UID: "8bc986a7-9816-4138-b597-5569cc44f31e"). InnerVolumeSpecName "kube-api-access-qdjjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:25.958694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.958669 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdjjp\" (UniqueName: \"kubernetes.io/projected/8bc986a7-9816-4138-b597-5569cc44f31e-kube-api-access-qdjjp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:25.958694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:25.958694 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bc986a7-9816-4138-b597-5569cc44f31e-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:26.430004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.429918 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:26.541137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.541107 4573 generic.go:356] "Generic (PLEG): container finished" podID="3f5e8bd2-36db-4129-a074-83b8617ddfcf" containerID="ed8941ff71d896eeeae2771f9b1108954284b9571bd0f7d476aea8652ccc999f" exitCode=0 Nov 01 16:03:26.541577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.541204 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" event={"ID":"3f5e8bd2-36db-4129-a074-83b8617ddfcf","Type":"ContainerDied","Data":"ed8941ff71d896eeeae2771f9b1108954284b9571bd0f7d476aea8652ccc999f"} Nov 01 16:03:26.543438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.542826 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" Nov 01 16:03:26.543438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.542834 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8" event={"ID":"8bc986a7-9816-4138-b597-5569cc44f31e","Type":"ContainerDied","Data":"fc53a65acbc927b269b6a334f65208b14077a5303a2746eb3db82aa53e7b9aa1"} Nov 01 16:03:26.543438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.542888 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc53a65acbc927b269b6a334f65208b14077a5303a2746eb3db82aa53e7b9aa1" Nov 01 16:03:26.746837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.746703 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:26.871216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.871182 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-xks-ca\") pod \"775c8e45-c357-46f1-acc4-01adc262ff1d\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " Nov 01 16:03:26.871368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.871296 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775c8e45-c357-46f1-acc4-01adc262ff1d-kube-api-access-7fkww\" (UniqueName: \"kubernetes.io/projected/775c8e45-c357-46f1-acc4-01adc262ff1d-kube-api-access-7fkww\") pod \"775c8e45-c357-46f1-acc4-01adc262ff1d\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " Nov 01 16:03:26.871486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.871472 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-openshift-ca\") pod \"775c8e45-c357-46f1-acc4-01adc262ff1d\" (UID: \"775c8e45-c357-46f1-acc4-01adc262ff1d\") " Nov 01 16:03:26.871820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.871790 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-openshift-ca" pod "775c8e45-c357-46f1-acc4-01adc262ff1d" (UID: "775c8e45-c357-46f1-acc4-01adc262ff1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:26.872040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.872024 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:26.873282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.873257 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-xks-ca" pod "775c8e45-c357-46f1-acc4-01adc262ff1d" (UID: "775c8e45-c357-46f1-acc4-01adc262ff1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:26.873487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.873466 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775c8e45-c357-46f1-acc4-01adc262ff1d-kube-api-access-7fkww" pod "775c8e45-c357-46f1-acc4-01adc262ff1d" (UID: "775c8e45-c357-46f1-acc4-01adc262ff1d"). InnerVolumeSpecName "kube-api-access-7fkww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:26.973548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.973482 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c8e45-c357-46f1-acc4-01adc262ff1d-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:26.973548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:26.973511 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fkww\" (UniqueName: \"kubernetes.io/projected/775c8e45-c357-46f1-acc4-01adc262ff1d-kube-api-access-7fkww\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:27.155574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.155543 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:27.552198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.552168 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" Nov 01 16:03:27.552198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.552189 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664" event={"ID":"775c8e45-c357-46f1-acc4-01adc262ff1d","Type":"ContainerDied","Data":"10a75df4c2b21b9646c34a2c5fa776c0e3f7804b8c1d68f2ea484c921bcaec82"} Nov 01 16:03:27.552625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.552244 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10a75df4c2b21b9646c34a2c5fa776c0e3f7804b8c1d68f2ea484c921bcaec82" Nov 01 16:03:27.725555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.725527 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:27.798465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.798433 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:27.888506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.888468 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-xks-ca\") pod \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " Nov 01 16:03:27.889254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.889226 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-openshift-ca\") pod \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " Nov 01 16:03:27.895779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.889614 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5e8bd2-36db-4129-a074-83b8617ddfcf-kube-api-access-qfzmb\" (UniqueName: \"kubernetes.io/projected/3f5e8bd2-36db-4129-a074-83b8617ddfcf-kube-api-access-qfzmb\") pod \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\" (UID: \"3f5e8bd2-36db-4129-a074-83b8617ddfcf\") " Nov 01 16:03:27.899225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.892563 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-openshift-ca" pod "3f5e8bd2-36db-4129-a074-83b8617ddfcf" (UID: "3f5e8bd2-36db-4129-a074-83b8617ddfcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:27.908711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.904983 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-xks-ca" pod "3f5e8bd2-36db-4129-a074-83b8617ddfcf" (UID: "3f5e8bd2-36db-4129-a074-83b8617ddfcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:27.908711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.906322 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5e8bd2-36db-4129-a074-83b8617ddfcf-kube-api-access-qfzmb" pod "3f5e8bd2-36db-4129-a074-83b8617ddfcf" (UID: "3f5e8bd2-36db-4129-a074-83b8617ddfcf"). InnerVolumeSpecName "kube-api-access-qfzmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:27.998499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.998466 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:27.998499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.998494 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5e8bd2-36db-4129-a074-83b8617ddfcf-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:27.998698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:27.998508 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzmb\" (UniqueName: \"kubernetes.io/projected/3f5e8bd2-36db-4129-a074-83b8617ddfcf-kube-api-access-qfzmb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:28.481586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.481553 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:28.609732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.609696 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" event={"ID":"3f5e8bd2-36db-4129-a074-83b8617ddfcf","Type":"ContainerDied","Data":"8c61d6182884d74a9ab86272c479839d844816ac8a129837c1c9ff731b969859"} Nov 01 16:03:28.610267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.609790 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c61d6182884d74a9ab86272c479839d844816ac8a129837c1c9ff731b969859" Nov 01 16:03:28.610267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.609880 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f" Nov 01 16:03:28.917241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.917204 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf"] Nov 01 16:03:28.918031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918009 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8bc986a7-9816-4138-b597-5569cc44f31e" containerName="revoke-keys" Nov 01 16:03:28.918031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918029 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bc986a7-9816-4138-b597-5569cc44f31e" containerName="revoke-keys" Nov 01 16:03:28.918219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918057 4573 cpu_manager.go:414] "containerMap: removing container" podUID="775c8e45-c357-46f1-acc4-01adc262ff1d" containerName="revoke-keys" Nov 01 16:03:28.918219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918065 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775c8e45-c357-46f1-acc4-01adc262ff1d" containerName="revoke-keys" Nov 01 16:03:28.918219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918099 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f08d1fd8-46ba-4c0d-bcc3-270fe5582c81" containerName="revoke-keys" Nov 01 16:03:28.918219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918108 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f08d1fd8-46ba-4c0d-bcc3-270fe5582c81" containerName="revoke-keys" Nov 01 16:03:28.918219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918127 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3f5e8bd2-36db-4129-a074-83b8617ddfcf" containerName="revoke-keys" Nov 01 16:03:28.918219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918134 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5e8bd2-36db-4129-a074-83b8617ddfcf" containerName="revoke-keys" Nov 01 16:03:28.918508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918327 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f08d1fd8-46ba-4c0d-bcc3-270fe5582c81" containerName="revoke-keys" Nov 01 16:03:28.918508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918343 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5e8bd2-36db-4129-a074-83b8617ddfcf" containerName="revoke-keys" Nov 01 16:03:28.918508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918361 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bc986a7-9816-4138-b597-5569cc44f31e" containerName="revoke-keys" Nov 01 16:03:28.918508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.918373 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="775c8e45-c357-46f1-acc4-01adc262ff1d" containerName="revoke-keys" Nov 01 16:03:28.923084 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.923061 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:28.926934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.926914 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:03:28.927107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.926910 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:03:28.928946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:28.928926 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf"] Nov 01 16:03:29.012790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.012738 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.012989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.012926 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.012989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.012975 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgww5\" (UniqueName: \"kubernetes.io/projected/a2ac7cd9-31bf-44d2-8242-992a00f08e56-kube-api-access-wgww5\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.114226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.114190 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgww5\" (UniqueName: \"kubernetes.io/projected/a2ac7cd9-31bf-44d2-8242-992a00f08e56-kube-api-access-wgww5\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.114417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.114402 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.114544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.114528 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.115173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.115148 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.116799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.116781 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.123690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.123670 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgww5\" (UniqueName: \"kubernetes.io/projected/a2ac7cd9-31bf-44d2-8242-992a00f08e56-kube-api-access-wgww5\") pod \"maas-api-revoke-sub-cb43ceb2-9nnxf\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.242612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.242533 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:29.379886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.379836 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:29.530320 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:29.530291 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ac7cd9_31bf_44d2_8242_992a00f08e56.slice/crio-234be2b7d38995002bf102480acb2b4866f349a060dbcba162b8245c36ad24fe WatchSource:0}: Error finding container 234be2b7d38995002bf102480acb2b4866f349a060dbcba162b8245c36ad24fe: Status 404 returned error can't find the container with id 234be2b7d38995002bf102480acb2b4866f349a060dbcba162b8245c36ad24fe Nov 01 16:03:29.532253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.532229 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf"] Nov 01 16:03:29.620179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:29.620145 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" event={"ID":"a2ac7cd9-31bf-44d2-8242-992a00f08e56","Type":"ContainerStarted","Data":"234be2b7d38995002bf102480acb2b4866f349a060dbcba162b8245c36ad24fe"} Nov 01 16:03:30.222356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:30.222258 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:30.630830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:30.630798 4573 generic.go:356] "Generic (PLEG): container finished" podID="a2ac7cd9-31bf-44d2-8242-992a00f08e56" containerID="3adfef270fecfd4281ef0abe6ece49303d33f81f26ef04d7b27dd6505c1f9a3c" exitCode=0 Nov 01 16:03:30.631250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:30.630887 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" event={"ID":"a2ac7cd9-31bf-44d2-8242-992a00f08e56","Type":"ContainerDied","Data":"3adfef270fecfd4281ef0abe6ece49303d33f81f26ef04d7b27dd6505c1f9a3c"} Nov 01 16:03:30.666207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:30.666175 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:31.075591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.075552 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:31.815200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.815171 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:31.819709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.819682 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm"] Nov 01 16:03:31.820399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.820385 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a2ac7cd9-31bf-44d2-8242-992a00f08e56" containerName="revoke-keys" Nov 01 16:03:31.820450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.820400 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ac7cd9-31bf-44d2-8242-992a00f08e56" containerName="revoke-keys" Nov 01 16:03:31.820524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.820516 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ac7cd9-31bf-44d2-8242-992a00f08e56" containerName="revoke-keys" Nov 01 16:03:31.824383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.824359 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.831274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.831250 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm"] Nov 01 16:03:31.869946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.869916 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-openshift-ca\") pod \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " Nov 01 16:03:31.870114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.870061 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-xks-ca\") pod \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " Nov 01 16:03:31.870239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.870221 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ac7cd9-31bf-44d2-8242-992a00f08e56-kube-api-access-wgww5\" (UniqueName: \"kubernetes.io/projected/a2ac7cd9-31bf-44d2-8242-992a00f08e56-kube-api-access-wgww5\") pod \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\" (UID: \"a2ac7cd9-31bf-44d2-8242-992a00f08e56\") " Nov 01 16:03:31.870331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.870309 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-openshift-ca" pod "a2ac7cd9-31bf-44d2-8242-992a00f08e56" (UID: "a2ac7cd9-31bf-44d2-8242-992a00f08e56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:31.870592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.870567 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895e104b-6d5d-4d63-88f8-055593234253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.870839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.870812 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cb2\" (UniqueName: \"kubernetes.io/projected/895e104b-6d5d-4d63-88f8-055593234253-kube-api-access-58cb2\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.871117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.871086 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895e104b-6d5d-4d63-88f8-055593234253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.871312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.871294 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:31.872326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.872305 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-xks-ca" pod "a2ac7cd9-31bf-44d2-8242-992a00f08e56" (UID: "a2ac7cd9-31bf-44d2-8242-992a00f08e56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:31.872490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.872473 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ac7cd9-31bf-44d2-8242-992a00f08e56-kube-api-access-wgww5" pod "a2ac7cd9-31bf-44d2-8242-992a00f08e56" (UID: "a2ac7cd9-31bf-44d2-8242-992a00f08e56"). InnerVolumeSpecName "kube-api-access-wgww5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:31.936641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.936564 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:31.972657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.972623 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895e104b-6d5d-4d63-88f8-055593234253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.972843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.972764 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895e104b-6d5d-4d63-88f8-055593234253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.972938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.972924 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cb2\" (UniqueName: \"kubernetes.io/projected/895e104b-6d5d-4d63-88f8-055593234253-kube-api-access-58cb2\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.973060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.973045 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgww5\" (UniqueName: \"kubernetes.io/projected/a2ac7cd9-31bf-44d2-8242-992a00f08e56-kube-api-access-wgww5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:31.973112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.973068 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2ac7cd9-31bf-44d2-8242-992a00f08e56-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:31.973264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.973246 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895e104b-6d5d-4d63-88f8-055593234253-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.974985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.974961 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895e104b-6d5d-4d63-88f8-055593234253-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:31.979950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:31.979928 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cb2\" (UniqueName: \"kubernetes.io/projected/895e104b-6d5d-4d63-88f8-055593234253-kube-api-access-58cb2\") pod \"maas-api-revoke-sub-bca74c7b-xv4qm\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:32.144155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:32.144123 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:32.430739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:32.430705 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm"] Nov 01 16:03:32.651460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:32.651424 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" event={"ID":"895e104b-6d5d-4d63-88f8-055593234253","Type":"ContainerStarted","Data":"40a9cec77e631b625cb0ebe110cf8bb64728065709d04812a68e954f16c33991"} Nov 01 16:03:32.653011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:32.652984 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" Nov 01 16:03:32.653124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:32.653008 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf" event={"ID":"a2ac7cd9-31bf-44d2-8242-992a00f08e56","Type":"ContainerDied","Data":"234be2b7d38995002bf102480acb2b4866f349a060dbcba162b8245c36ad24fe"} Nov 01 16:03:32.653124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:32.653056 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="234be2b7d38995002bf102480acb2b4866f349a060dbcba162b8245c36ad24fe" Nov 01 16:03:32.878184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:32.878158 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:33.107841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.107810 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578"] Nov 01 16:03:33.114704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.114616 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.121113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.121083 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578"] Nov 01 16:03:33.195633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.195556 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.195633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.195593 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.195832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.195712 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-274mv\" (UniqueName: \"kubernetes.io/projected/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-kube-api-access-274mv\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.297718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.297688 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.297859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.297730 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.298294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.297938 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-274mv\" (UniqueName: \"kubernetes.io/projected/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-kube-api-access-274mv\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.300119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.300097 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.301053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.301032 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.306225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.306199 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-274mv\" (UniqueName: \"kubernetes.io/projected/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-kube-api-access-274mv\") pod \"maas-api-revoke-sub-6961a50d-jm578\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.435775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.434830 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:33.747825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.737976 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:33.747825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.745097 4573 generic.go:356] "Generic (PLEG): container finished" podID="895e104b-6d5d-4d63-88f8-055593234253" containerID="ff6ec7dc44f7585845522966d418ce8beead9a3f32e228b155f280fc26b8e982" exitCode=0 Nov 01 16:03:33.747825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:33.745294 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" event={"ID":"895e104b-6d5d-4d63-88f8-055593234253","Type":"ContainerDied","Data":"ff6ec7dc44f7585845522966d418ce8beead9a3f32e228b155f280fc26b8e982"} Nov 01 16:03:34.632803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:34.632329 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:34.738782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:34.737875 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:34.741783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:34.740319 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.154882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.144587 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.154882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.144784 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:03:35.281139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.273851 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" event={"ID":"8419a141-d63f-4bc4-b2ae-bbb424cb2e86","Type":"ContainerStarted","Data":"8dab47e8b7716e304125fd8fc51ba52249fee507fc050ed3fd6a93063b1b493c"} Nov 01 16:03:35.326802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.311229 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.425592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.425346 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578"] Nov 01 16:03:35.506064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.504856 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.575587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.558246 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs"] Nov 01 16:03:35.577482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.577437 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.600701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.600668 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:35.709911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.701975 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:35.709911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.702067 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:35.709911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.702157 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw85d\" (UniqueName: \"kubernetes.io/projected/227419a0-7ea0-40df-a1c1-68228db79e84-kube-api-access-tw85d\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:35.752798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.746932 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.752798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.748081 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs"] Nov 01 16:03:35.752798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.748195 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:03:35.800779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.763644 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.884777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.802639 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:03:35.903942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.812431 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:35.903942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.901047 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw85d\" (UniqueName: \"kubernetes.io/projected/227419a0-7ea0-40df-a1c1-68228db79e84-kube-api-access-tw85d\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:35.903942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.901374 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:35.903942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.881679 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:35.903942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.813264 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:35.903942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:35.814403 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.040914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.040802 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.337845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.302690 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.337845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.303837 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:36.337845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.304458 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:03:36.337845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.304517 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:03:36.337845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.311457 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.384784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.347928 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:03:36.441570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.441534 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw85d\" (UniqueName: \"kubernetes.io/projected/227419a0-7ea0-40df-a1c1-68228db79e84-kube-api-access-tw85d\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:36.519779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.494389 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2e2dfd-zmqgs\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:36.554776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.521718 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.621196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.600824 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:36.621196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.601415 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:36.870777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.808411 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:36.870777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.842236 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:36.952785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.949792 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:03:36.967946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:36.967900 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:03:37.008810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.006978 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.347274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.342115 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:03:37.347274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.342467 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.347274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.342798 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.488945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.488907 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:37.489219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.489188 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.572832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.532118 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:37.613776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.613012 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.711778 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:37.676195 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.213s" Nov 01 16:03:37.711778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.676328 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:03:37.711778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.676370 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:03:37.711778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.676429 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:03:37.711778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.676445 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" event={"ID":"8419a141-d63f-4bc4-b2ae-bbb424cb2e86","Type":"ContainerStarted","Data":"cea987321f075899d77879c1f53b46e4a79fc298bb2853d4f4967d0129b90480"} Nov 01 16:03:37.711778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.676518 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:03:37.819810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.817884 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:37.846269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.844881 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.892983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.892946 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:37.902467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.902423 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.946835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.929799 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:37.946835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.929863 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.957800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.947325 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:37.957800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.947388 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:37.957800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.947455 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:37.957800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.947478 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.021803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.985703 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.021803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:37.986091 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.083080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.083042 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.083338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.083309 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.129473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.127844 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.129473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.127904 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.341771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.340891 4573 patch_prober.go:29] interesting pod/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.341771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.340982 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.343901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.343861 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.344417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.344386 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.344861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.344836 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.454701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.442407 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" podStartSLOduration=5.442387492 podStartE2EDuration="5.442387492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:37.636119187 +0000 UTC m=+955.339285126" watchObservedRunningTime="2026-11-01 16:03:38.442387492 +0000 UTC m=+956.145553429" Nov 01 16:03:38.662779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.588298 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:38.662779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.613846 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.662779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.641864 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:38.662779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.641929 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.818406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.818302 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:38.845542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.845496 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.926781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.898898 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:38.926781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:38.899544 4573 generic.go:356] "Generic (PLEG): container finished" podID="8419a141-d63f-4bc4-b2ae-bbb424cb2e86" containerID="cea987321f075899d77879c1f53b46e4a79fc298bb2853d4f4967d0129b90480" exitCode=0 Nov 01 16:03:39.037853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.037808 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:03:39.054286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.038109 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.366088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.250987 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.366088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.332156 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:39.366088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.332232 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.618992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.617905 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:39.672029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.671984 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:39.676774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.672821 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:39.697776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.672368 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:03:40.104457 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.672394 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:03:40.132817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.672426 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:03:40.132817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.672309 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.133195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.941796 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:40.133398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.133364 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.136411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.941904 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:03:40.195996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:39.945804 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:40.195996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.040430 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Nov 01 16:03:40.195996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.064965 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:03:40.251955 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:40.251912 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.665s" Nov 01 16:03:40.252238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.252208 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" event={"ID":"8419a141-d63f-4bc4-b2ae-bbb424cb2e86","Type":"ContainerDied","Data":"cea987321f075899d77879c1f53b46e4a79fc298bb2853d4f4967d0129b90480"} Nov 01 16:03:40.252643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.252613 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:03:40.407140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.407103 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:40.407399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.407352 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.654773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.653963 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.693829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.689695 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:40.936778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.888023 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/895e104b-6d5d-4d63-88f8-055593234253-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895e104b-6d5d-4d63-88f8-055593234253-maas-api-xks-ca\") pod \"895e104b-6d5d-4d63-88f8-055593234253\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " Nov 01 16:03:40.975137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.893919 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" podUID="2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:40.975456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.975431 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895e104b-6d5d-4d63-88f8-055593234253-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895e104b-6d5d-4d63-88f8-055593234253-maas-api-openshift-ca\") pod \"895e104b-6d5d-4d63-88f8-055593234253\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " Nov 01 16:03:40.976273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.976249 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895e104b-6d5d-4d63-88f8-055593234253-kube-api-access-58cb2\" (UniqueName: \"kubernetes.io/projected/895e104b-6d5d-4d63-88f8-055593234253-kube-api-access-58cb2\") pod \"895e104b-6d5d-4d63-88f8-055593234253\" (UID: \"895e104b-6d5d-4d63-88f8-055593234253\") " Nov 01 16:03:41.069811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.975913 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895e104b-6d5d-4d63-88f8-055593234253-maas-api-openshift-ca" pod "895e104b-6d5d-4d63-88f8-055593234253" (UID: "895e104b-6d5d-4d63-88f8-055593234253"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:41.069811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:40.982291 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/895e104b-6d5d-4d63-88f8-055593234253-maas-api-xks-ca" pod "895e104b-6d5d-4d63-88f8-055593234253" (UID: "895e104b-6d5d-4d63-88f8-055593234253"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:41.100017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.084710 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" event={"ID":"895e104b-6d5d-4d63-88f8-055593234253","Type":"ContainerDied","Data":"40a9cec77e631b625cb0ebe110cf8bb64728065709d04812a68e954f16c33991"} Nov 01 16:03:41.100017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.084801 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40a9cec77e631b625cb0ebe110cf8bb64728065709d04812a68e954f16c33991" Nov 01 16:03:41.100017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.084825 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs"] Nov 01 16:03:41.100017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.086283 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/895e104b-6d5d-4d63-88f8-055593234253-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:41.100017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.086312 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/895e104b-6d5d-4d63-88f8-055593234253-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:41.135845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.103879 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.135845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.112844 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.135845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.113207 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.138795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.136572 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.138795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.136731 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.329839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.206858 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.329839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.252936 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895e104b-6d5d-4d63-88f8-055593234253-kube-api-access-58cb2" pod "895e104b-6d5d-4d63-88f8-055593234253" (UID: "895e104b-6d5d-4d63-88f8-055593234253"). InnerVolumeSpecName "kube-api-access-58cb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:41.329839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.253183 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.329839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.253545 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:41.344833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.321819 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cb2\" (UniqueName: \"kubernetes.io/projected/895e104b-6d5d-4d63-88f8-055593234253-kube-api-access-58cb2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:41.466891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.466847 4573 patch_prober.go:29] interesting pod/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:03:41.467188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.467159 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:41.672203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:41.672160 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:42.093787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.090353 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm" Nov 01 16:03:42.183948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.113864 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" event={"ID":"227419a0-7ea0-40df-a1c1-68228db79e84","Type":"ContainerStarted","Data":"8ae54590bc6ec4b01c4a4d4462432cfed16f2a910833764e7dd7d613baaa7205"} Nov 01 16:03:42.183948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.115327 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:03:42.183948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.116286 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.183948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.116549 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.189536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.187029 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.189536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.187347 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:03:42.231481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.229172 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:42.256901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.256854 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:03:42.260774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.257145 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:03:42.670830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.558478 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 01 16:03:42.744988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.740791 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:03:42.823012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:42.820012 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.117782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.116122 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.117782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.116677 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.117782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.116959 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:03:43.205039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.204994 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.218785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.205934 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.231271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.231171 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:43.232123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.231563 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.258409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.257864 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.291813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.291707 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:03:43.308856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.306804 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:43.328783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.323471 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" event={"ID":"227419a0-7ea0-40df-a1c1-68228db79e84","Type":"ContainerStarted","Data":"44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3"} Nov 01 16:03:43.341636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.341611 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:03:43.343286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.343267 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:03:43.441674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.441642 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-xks-ca\") pod \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " Nov 01 16:03:43.472969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.472916 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-openshift-ca\") pod \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " Nov 01 16:03:43.473161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.473030 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-kube-api-access-274mv\" (UniqueName: \"kubernetes.io/projected/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-kube-api-access-274mv\") pod \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\" (UID: \"8419a141-d63f-4bc4-b2ae-bbb424cb2e86\") " Nov 01 16:03:43.474374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.474334 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-xks-ca" pod "8419a141-d63f-4bc4-b2ae-bbb424cb2e86" (UID: "8419a141-d63f-4bc4-b2ae-bbb424cb2e86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:43.523267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.522997 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-openshift-ca" pod "8419a141-d63f-4bc4-b2ae-bbb424cb2e86" (UID: "8419a141-d63f-4bc4-b2ae-bbb424cb2e86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:43.524805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.523864 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:03:43.528229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.527462 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:03:43.528229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.527522 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:03:43.539003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.538977 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-kube-api-access-274mv" pod "8419a141-d63f-4bc4-b2ae-bbb424cb2e86" (UID: "8419a141-d63f-4bc4-b2ae-bbb424cb2e86"). InnerVolumeSpecName "kube-api-access-274mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:43.544462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.544409 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:03:43.545023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.545008 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:03:43.545219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.545207 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:03:43.571972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.571943 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" Nov 01 16:03:43.572451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.572419 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578" event={"ID":"8419a141-d63f-4bc4-b2ae-bbb424cb2e86","Type":"ContainerDied","Data":"8dab47e8b7716e304125fd8fc51ba52249fee507fc050ed3fd6a93063b1b493c"} Nov 01 16:03:43.572683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.572665 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dab47e8b7716e304125fd8fc51ba52249fee507fc050ed3fd6a93063b1b493c" Nov 01 16:03:43.577223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.577197 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:43.577337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.577229 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:43.577337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.577250 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-274mv\" (UniqueName: \"kubernetes.io/projected/8419a141-d63f-4bc4-b2ae-bbb424cb2e86-kube-api-access-274mv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:43.585439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.585416 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:03:43.594673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:43.594607 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" podStartSLOduration=9.59459027 podStartE2EDuration="9.59459027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:43.594328853 +0000 UTC m=+961.297494814" watchObservedRunningTime="2026-11-01 16:03:43.59459027 +0000 UTC m=+961.297756211" Nov 01 16:03:44.404705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:44.404669 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:44.583863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:44.583825 4573 generic.go:356] "Generic (PLEG): container finished" podID="227419a0-7ea0-40df-a1c1-68228db79e84" containerID="44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3" exitCode=0 Nov 01 16:03:44.584088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:44.583863 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" event={"ID":"227419a0-7ea0-40df-a1c1-68228db79e84","Type":"ContainerDied","Data":"44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3"} Nov 01 16:03:45.170706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.170673 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:45.772288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.772261 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:45.924227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.924195 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/227419a0-7ea0-40df-a1c1-68228db79e84-kube-api-access-tw85d\" (UniqueName: \"kubernetes.io/projected/227419a0-7ea0-40df-a1c1-68228db79e84-kube-api-access-tw85d\") pod \"227419a0-7ea0-40df-a1c1-68228db79e84\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " Nov 01 16:03:45.924431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.924294 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-xks-ca\") pod \"227419a0-7ea0-40df-a1c1-68228db79e84\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " Nov 01 16:03:45.924512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.924448 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-openshift-ca\") pod \"227419a0-7ea0-40df-a1c1-68228db79e84\" (UID: \"227419a0-7ea0-40df-a1c1-68228db79e84\") " Nov 01 16:03:45.924831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.924809 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-openshift-ca" pod "227419a0-7ea0-40df-a1c1-68228db79e84" (UID: "227419a0-7ea0-40df-a1c1-68228db79e84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:03:45.925028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.925013 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:45.926347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.926327 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-xks-ca" pod "227419a0-7ea0-40df-a1c1-68228db79e84" (UID: "227419a0-7ea0-40df-a1c1-68228db79e84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:03:45.926347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:45.926333 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/227419a0-7ea0-40df-a1c1-68228db79e84-kube-api-access-tw85d" pod "227419a0-7ea0-40df-a1c1-68228db79e84" (UID: "227419a0-7ea0-40df-a1c1-68228db79e84"). InnerVolumeSpecName "kube-api-access-tw85d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:03:46.018094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.018061 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:46.026897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.026867 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw85d\" (UniqueName: \"kubernetes.io/projected/227419a0-7ea0-40df-a1c1-68228db79e84-kube-api-access-tw85d\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:46.026897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.026892 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/227419a0-7ea0-40df-a1c1-68228db79e84-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:03:46.372839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.372792 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:46.604051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.604012 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" event={"ID":"227419a0-7ea0-40df-a1c1-68228db79e84","Type":"ContainerDied","Data":"8ae54590bc6ec4b01c4a4d4462432cfed16f2a910833764e7dd7d613baaa7205"} Nov 01 16:03:46.604219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.604051 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs" Nov 01 16:03:46.604219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.604067 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ae54590bc6ec4b01c4a4d4462432cfed16f2a910833764e7dd7d613baaa7205" Nov 01 16:03:46.918997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.918674 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq"] Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.919847 4573 cpu_manager.go:414] "containerMap: removing container" podUID="895e104b-6d5d-4d63-88f8-055593234253" containerName="revoke-keys" Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.919868 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895e104b-6d5d-4d63-88f8-055593234253" containerName="revoke-keys" Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.919884 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8419a141-d63f-4bc4-b2ae-bbb424cb2e86" containerName="revoke-keys" Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.919893 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8419a141-d63f-4bc4-b2ae-bbb424cb2e86" containerName="revoke-keys" Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.919917 4573 cpu_manager.go:414] "containerMap: removing container" podUID="227419a0-7ea0-40df-a1c1-68228db79e84" containerName="revoke-keys" Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.919925 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="227419a0-7ea0-40df-a1c1-68228db79e84" containerName="revoke-keys" Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.920105 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="895e104b-6d5d-4d63-88f8-055593234253" containerName="revoke-keys" Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.920119 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8419a141-d63f-4bc4-b2ae-bbb424cb2e86" containerName="revoke-keys" Nov 01 16:03:46.920619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.920133 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="227419a0-7ea0-40df-a1c1-68228db79e84" containerName="revoke-keys" Nov 01 16:03:46.925623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.925593 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:46.946167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.937519 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-8624bae9-openshift-default-dockercfg-4vjzq\"" Nov 01 16:03:46.946167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.940472 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:46.946167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:46.946052 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq"] Nov 01 16:03:47.042574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.042540 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-data\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.042765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.042591 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-credential-socket\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.042863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.042767 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1770918f-862c-4189-9eef-1260c25f619d-istiod-ca-cert\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.042863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.042824 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-envoy\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.042964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.042890 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-certs\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.043018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.043002 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc2b5\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-kube-api-access-pc2b5\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.043082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.043041 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-socket\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.043082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.043057 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1770918f-862c-4189-9eef-1260c25f619d-istio-podinfo\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.043260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.043238 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-istio-token\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.075589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.075557 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.144862 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-certs\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.145007 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc2b5\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-kube-api-access-pc2b5\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.145063 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-socket\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.145088 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1770918f-862c-4189-9eef-1260c25f619d-istio-podinfo\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.145232 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-istio-token\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.145302 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-data\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.145341 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-credential-socket\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.145402 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1770918f-862c-4189-9eef-1260c25f619d-istiod-ca-cert\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.145445 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-envoy\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.147050 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-certs\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.147325 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-data\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.147935 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-credential-socket\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.148190 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-socket\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.151787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.148813 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1770918f-862c-4189-9eef-1260c25f619d-istiod-ca-cert\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.207824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.202449 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc2b5\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-kube-api-access-pc2b5\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.207824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.202831 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:47.207824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.203233 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-envoy\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.207824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.203932 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1770918f-862c-4189-9eef-1260c25f619d-istio-podinfo\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.234773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.214296 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-istio-token\") pod \"e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.255796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.254067 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:47.505987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.505899 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:47.576049 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:47.576015 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1770918f_862c_4189_9eef_1260c25f619d.slice/crio-2520de558a5fd39b23fcb231a50b7c5ffa3c38769b1bcd3cf1f147dd5861dae7 WatchSource:0}: Error finding container 2520de558a5fd39b23fcb231a50b7c5ffa3c38769b1bcd3cf1f147dd5861dae7: Status 404 returned error can't find the container with id 2520de558a5fd39b23fcb231a50b7c5ffa3c38769b1bcd3cf1f147dd5861dae7 Nov 01 16:03:47.578343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.578318 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq"] Nov 01 16:03:47.578803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.578741 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:47.578883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.578834 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:47.578883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.578864 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:47.625773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.616475 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" event={"ID":"1770918f-862c-4189-9eef-1260c25f619d","Type":"ContainerStarted","Data":"2520de558a5fd39b23fcb231a50b7c5ffa3c38769b1bcd3cf1f147dd5861dae7"} Nov 01 16:03:47.877037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:47.876928 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:48.452118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:48.452087 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:48.560761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:48.560716 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:48.627147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:48.627116 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" event={"ID":"1770918f-862c-4189-9eef-1260c25f619d","Type":"ContainerStarted","Data":"fbeeee9817eaad016ee7637ef57c041ab0072df35bdb05fe95ca28e1f76d5eba"} Nov 01 16:03:48.662661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:48.662609 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" podStartSLOduration=2.66259485 podStartE2EDuration="2.66259485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:48.659263752 +0000 UTC m=+966.362429691" watchObservedRunningTime="2026-11-01 16:03:48.66259485 +0000 UTC m=+966.365760845" Nov 01 16:03:48.696872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:48.696842 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:48.808249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:48.808160 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:49.194119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:49.194084 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:49.255381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:49.255350 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:49.260513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:49.260489 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:49.326774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:49.325710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:49.637892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:49.637862 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:49.639119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:49.639091 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:03:49.959036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:49.958950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:50.103699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:50.103663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:50.204595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:50.204563 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:50.988323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:50.988292 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:51.322945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.322855 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:51.433285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.433252 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:51.437534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.437509 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s"] Nov 01 16:03:51.443140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.443113 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:51.449582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.449542 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8624bae9\"" Nov 01 16:03:51.449732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.449651 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8624bae9-dockercfg-vz8mz\"" Nov 01 16:03:51.453227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.453204 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s"] Nov 01 16:03:51.480193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.480159 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4vv\" (UniqueName: \"kubernetes.io/projected/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-kube-api-access-6k4vv\") pod \"payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s\" (UID: \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:51.480517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.480491 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s\" (UID: \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:51.537402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.537355 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:51.597926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.582456 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4vv\" (UniqueName: \"kubernetes.io/projected/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-kube-api-access-6k4vv\") pod \"payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s\" (UID: \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:51.597926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.582903 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s\" (UID: \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:51.597926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.584051 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-plugins-config-volume\") pod \"payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s\" (UID: \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:51.640555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.640515 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4vv\" (UniqueName: \"kubernetes.io/projected/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-kube-api-access-6k4vv\") pod \"payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s\" (UID: \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\") " pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:51.657626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.657594 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:51.767074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:51.767038 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:52.087080 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:52.087040 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda757c981_efd8_45bb_8b5b_b1754ed1c6ad.slice/crio-2b133b55a5e236b4057a92b17a112bd8e0ce62b55171e56581a8d628a9b98008 WatchSource:0}: Error finding container 2b133b55a5e236b4057a92b17a112bd8e0ce62b55171e56581a8d628a9b98008: Status 404 returned error can't find the container with id 2b133b55a5e236b4057a92b17a112bd8e0ce62b55171e56581a8d628a9b98008 Nov 01 16:03:52.088024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:52.087999 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s"] Nov 01 16:03:52.376922 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:52.376885 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:03:52.434278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:52.434248 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:52.670370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:52.670286 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:52.726274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:52.726240 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" event={"ID":"a757c981-efd8-45bb-8b5b-b1754ed1c6ad","Type":"ContainerStarted","Data":"295185b6e8b31d1f5d16f66f0db9b9bc8afd49c68bb13e8f140e5d76d4144b49"} Nov 01 16:03:52.726443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:52.726295 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" event={"ID":"a757c981-efd8-45bb-8b5b-b1754ed1c6ad","Type":"ContainerStarted","Data":"2b133b55a5e236b4057a92b17a112bd8e0ce62b55171e56581a8d628a9b98008"} Nov 01 16:03:52.726443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:52.726332 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:03:52.742930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:52.742881 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" podStartSLOduration=1.742864167 podStartE2EDuration="1.742864167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:52.742480599 +0000 UTC m=+970.445646535" watchObservedRunningTime="2026-11-01 16:03:52.742864167 +0000 UTC m=+970.446030106" Nov 01 16:03:52.746783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:52.746737 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:53.116289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:53.116256 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:53.217120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:53.217085 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:53.532965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:53.532875 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:54.355294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:54.355264 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:54.368407 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:54.368379 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:03:54.468728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:54.468666 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:54.615476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:54.615442 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:54.861621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:54.861221 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:55.213417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.213382 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm"] Nov 01 16:03:55.220231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.220205 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.229200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.229165 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ec6894c9-openshift-default-dockercfg-v8zw2\"" Nov 01 16:03:55.233341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.233312 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm"] Nov 01 16:03:55.257174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.257134 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:55.324463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.317420 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-token\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.331880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.331838 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-podinfo\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.332097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.332017 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzqz9\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-kube-api-access-fzqz9\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.332097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.332086 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-credential-socket\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.332228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.332115 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-certs\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.332228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.332154 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b3e5d-7d63-468e-ae3a-0f9aec607585-istiod-ca-cert\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.332228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.332206 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-envoy\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.332405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.332259 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-data\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.332405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.332349 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-socket\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.418971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.418849 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.435925 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-token\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.435975 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-podinfo\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.436093 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzqz9\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-kube-api-access-fzqz9\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.436152 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-credential-socket\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.436181 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-certs\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.436215 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b3e5d-7d63-468e-ae3a-0f9aec607585-istiod-ca-cert\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.436261 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-envoy\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.436308 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-data\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.436368 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-socket\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.454825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.436904 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-socket\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.476560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.472461 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-credential-socket\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.476560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.472725 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-certs\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.476560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.473376 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b3e5d-7d63-468e-ae3a-0f9aec607585-istiod-ca-cert\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.476560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.476413 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-envoy\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.476930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.476690 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-data\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.521570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.499630 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-podinfo\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.543348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.528785 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-token\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.544284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.544257 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzqz9\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-kube-api-access-fzqz9\") pod \"e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.568419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.568388 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:55.614680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.612942 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:55.829838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.828215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:55.920029 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:03:55.920000 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod221b3e5d_7d63_468e_ae3a_0f9aec607585.slice/crio-ac797f514f1f00769e80ca82ee7f9de95623af842fd591d9dd4be98a316f64b9 WatchSource:0}: Error finding container ac797f514f1f00769e80ca82ee7f9de95623af842fd591d9dd4be98a316f64b9: Status 404 returned error can't find the container with id ac797f514f1f00769e80ca82ee7f9de95623af842fd591d9dd4be98a316f64b9 Nov 01 16:03:55.925274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.925222 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:55.925391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.925321 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:55.925391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.925367 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:03:55.925868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.925844 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm"] Nov 01 16:03:55.940060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:55.940025 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:56.068853 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:56.068809 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:03:56.069146 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:56.069115 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:03:56.467602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:56.467571 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:56.819654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:56.819565 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" event={"ID":"221b3e5d-7d63-468e-ae3a-0f9aec607585","Type":"ContainerStarted","Data":"83e5ade1142854dd688ed9806c224f2a13436a62075b90dba5d512b9bd7dac50"} Nov 01 16:03:56.819836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:56.819664 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" event={"ID":"221b3e5d-7d63-468e-ae3a-0f9aec607585","Type":"ContainerStarted","Data":"ac797f514f1f00769e80ca82ee7f9de95623af842fd591d9dd4be98a316f64b9"} Nov 01 16:03:56.839101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:56.839048 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" podStartSLOduration=1.839034195 podStartE2EDuration="1.839034195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:03:56.835904296 +0000 UTC m=+974.539070231" watchObservedRunningTime="2026-11-01 16:03:56.839034195 +0000 UTC m=+974.542200130" Nov 01 16:03:56.856157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:56.856117 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:57.210228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:57.210193 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:57.311938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:57.311899 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:57.461832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:57.460790 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:57.589832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:57.585767 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:57.633005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:57.632970 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:57.743738 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:03:57.743647 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:03:57.832566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:57.832533 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:57.833562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:57.833541 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:03:58.385303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:58.385266 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:58.487582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:58.487548 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:58.700494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:58.700410 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:59.570132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:59.570098 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:59.707271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:59.707237 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:03:59.806089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:03:59.806048 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:00.726927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:00.726886 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:01.287705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:01.287663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:01.387045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:01.387000 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:01.582378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:01.582288 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:01.691535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:01.691496 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:01.798561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:01.798491 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:01.915827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:01.915746 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:02.446600 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:02.446551 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:04:02.600058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.600017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:02.650211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.650173 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz"] Nov 01 16:04:02.655303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.655276 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:02.665706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.665631 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz"] Nov 01 16:04:02.666955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.666932 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-47e1515a-dockercfg-z6mbz\"" Nov 01 16:04:02.667234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.667074 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-47e1515a\"" Nov 01 16:04:02.680190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.680159 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf"] Nov 01 16:04:02.693977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.693939 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf"] Nov 01 16:04:02.694120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.694065 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:02.695686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.695650 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:02.803785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.803058 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sx87\" (UniqueName: \"kubernetes.io/projected/920c2275-5fde-431c-8c58-1ad35a704ef0-kube-api-access-9sx87\") pod \"payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz\" (UID: \"920c2275-5fde-431c-8c58-1ad35a704ef0\") " pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:02.803785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.803211 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920c2275-5fde-431c-8c58-1ad35a704ef0-plugins-config-volume\") pod \"payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz\" (UID: \"920c2275-5fde-431c-8c58-1ad35a704ef0\") " pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:02.803785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.803305 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk4lf\" (UniqueName: \"kubernetes.io/projected/3cd71d44-e9b5-4533-a65c-3956b09b75f0-kube-api-access-mk4lf\") pod \"payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf\" (UID: \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:02.803785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.803345 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd71d44-e9b5-4533-a65c-3956b09b75f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf\" (UID: \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:02.825932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.824083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:02.905875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.905787 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sx87\" (UniqueName: \"kubernetes.io/projected/920c2275-5fde-431c-8c58-1ad35a704ef0-kube-api-access-9sx87\") pod \"payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz\" (UID: \"920c2275-5fde-431c-8c58-1ad35a704ef0\") " pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:02.906091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.905930 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920c2275-5fde-431c-8c58-1ad35a704ef0-plugins-config-volume\") pod \"payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz\" (UID: \"920c2275-5fde-431c-8c58-1ad35a704ef0\") " pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:02.906091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.906027 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk4lf\" (UniqueName: \"kubernetes.io/projected/3cd71d44-e9b5-4533-a65c-3956b09b75f0-kube-api-access-mk4lf\") pod \"payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf\" (UID: \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:02.906091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.906079 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd71d44-e9b5-4533-a65c-3956b09b75f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf\" (UID: \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:02.928353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.907069 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd71d44-e9b5-4533-a65c-3956b09b75f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf\" (UID: \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:02.937801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.935677 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920c2275-5fde-431c-8c58-1ad35a704ef0-plugins-config-volume\") pod \"payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz\" (UID: \"920c2275-5fde-431c-8c58-1ad35a704ef0\") " pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:02.965830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.962274 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sx87\" (UniqueName: \"kubernetes.io/projected/920c2275-5fde-431c-8c58-1ad35a704ef0-kube-api-access-9sx87\") pod \"payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz\" (UID: \"920c2275-5fde-431c-8c58-1ad35a704ef0\") " pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:03.000857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.991833 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk4lf\" (UniqueName: \"kubernetes.io/projected/3cd71d44-e9b5-4533-a65c-3956b09b75f0-kube-api-access-mk4lf\") pod \"payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf\" (UID: \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:03.000857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:02.992120 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:03.016510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.014488 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:03.425433 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:04:03.425397 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod920c2275_5fde_431c_8c58_1ad35a704ef0.slice/crio-1925c2c4d14e64229c477b78cab8b0da90545cb5eeef729ac9618e07e601c511 WatchSource:0}: Error finding container 1925c2c4d14e64229c477b78cab8b0da90545cb5eeef729ac9618e07e601c511: Status 404 returned error can't find the container with id 1925c2c4d14e64229c477b78cab8b0da90545cb5eeef729ac9618e07e601c511 Nov 01 16:04:03.442321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.440953 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz"] Nov 01 16:04:03.637017 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:04:03.636988 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd71d44_e9b5_4533_a65c_3956b09b75f0.slice/crio-aaa7a40ddcae5f5e18ec6713641a10b3c96d4d25d18074d8e306e7a7c5b70eaf WatchSource:0}: Error finding container aaa7a40ddcae5f5e18ec6713641a10b3c96d4d25d18074d8e306e7a7c5b70eaf: Status 404 returned error can't find the container with id aaa7a40ddcae5f5e18ec6713641a10b3c96d4d25d18074d8e306e7a7c5b70eaf Nov 01 16:04:03.638941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.638917 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf"] Nov 01 16:04:03.661173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.661136 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:03.777856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.777701 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:04:03.989132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.989098 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" event={"ID":"3cd71d44-e9b5-4533-a65c-3956b09b75f0","Type":"ContainerStarted","Data":"381cf51c190802739c9be020a61259a9bb01eb42635b43fab360a62d383dfaa0"} Nov 01 16:04:03.989550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.989155 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" event={"ID":"3cd71d44-e9b5-4533-a65c-3956b09b75f0","Type":"ContainerStarted","Data":"aaa7a40ddcae5f5e18ec6713641a10b3c96d4d25d18074d8e306e7a7c5b70eaf"} Nov 01 16:04:03.989550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.989176 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:03.991090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.991052 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" event={"ID":"920c2275-5fde-431c-8c58-1ad35a704ef0","Type":"ContainerStarted","Data":"bfb18c5738901d36f31464373c5573f351e5b9691239c9b95dab599e7937a972"} Nov 01 16:04:03.991187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.991105 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" event={"ID":"920c2275-5fde-431c-8c58-1ad35a704ef0","Type":"ContainerStarted","Data":"1925c2c4d14e64229c477b78cab8b0da90545cb5eeef729ac9618e07e601c511"} Nov 01 16:04:03.991187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:03.991131 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:04.007809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:04.007768 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" podStartSLOduration=2.007741314 podStartE2EDuration="2.007741314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:04.002135453 +0000 UTC m=+981.705301393" watchObservedRunningTime="2026-11-01 16:04:04.007741314 +0000 UTC m=+981.710907265" Nov 01 16:04:04.022184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:04.022138 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" podStartSLOduration=2.022125366 podStartE2EDuration="2.022125366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:04.019027993 +0000 UTC m=+981.722193930" watchObservedRunningTime="2026-11-01 16:04:04.022125366 +0000 UTC m=+981.725291301" Nov 01 16:04:04.369309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:04.369276 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:05.059811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:05.059735 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:05.520677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:05.520646 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:05.591785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:05.591733 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:05.695344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:05.695313 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:06.451914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:06.451883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:06.574775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:06.571958 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:06.653380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:06.653338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:06.752901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:06.752823 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:07.368879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:07.368845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:08.147299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:08.147268 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:08.242121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:08.242090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:08.812742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:08.812706 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:04:09.043003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:09.042973 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:10.000136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:10.000100 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:10.084555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:10.084409 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:10.540621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:10.540484 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:11.289778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.286164 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt"] Nov 01 16:04:11.328102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.328072 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.482296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.422033 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:04:11.482296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.458968 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:04:11.486500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.486402 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt"] Nov 01 16:04:11.609833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.608017 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.609833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.608102 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.609833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.608424 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn462\" (UniqueName: \"kubernetes.io/projected/4975fc93-d65c-436c-a92e-035e40e2aaa3-kube-api-access-hn462\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.772368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.742293 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn462\" (UniqueName: \"kubernetes.io/projected/4975fc93-d65c-436c-a92e-035e40e2aaa3-kube-api-access-hn462\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.774945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.774902 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.775910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.775887 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.816013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.796531 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.816013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.815817 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn462\" (UniqueName: \"kubernetes.io/projected/4975fc93-d65c-436c-a92e-035e40e2aaa3-kube-api-access-hn462\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:11.855718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:11.840223 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-8624bae9-47slt\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:12.047828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.034882 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.073074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.073038 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:12.212027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.209655 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:04:12.276797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.274857 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:04:12.356842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.356433 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.359492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.357641 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.359492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.358558 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:04:12.416121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.415328 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.865213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.865157 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.875030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.874978 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:12.979345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:12.979287 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:04:13.042867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.042806 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded" Nov 01 16:04:13.322287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.322185 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:04:13.500221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.500171 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:13.681345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.561472 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded" Nov 01 16:04:13.681640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.599901 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:04:13.903771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.902804 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:04:13.934854 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:13.917029 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.446s" Nov 01 16:04:13.958006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.935242 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:04:14.002777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.992590 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:04:14.006774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.003102 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:04:14.008781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.007063 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.020773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.958708 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.037775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.033914 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:04:14.191868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.189891 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:04:14.204781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:13.991057 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:04:14.271836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.104987 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.281774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.107310 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:04:14.307777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.282687 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:04:14.479068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.432370 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:04:14.479068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.432426 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:04:14.479068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.432465 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:04:14.479068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.309740 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:14.479068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.337899 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:14.810903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.810861 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:04:14.928421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:14.928108 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.039233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.039125 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.039959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.039917 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:04:15.068067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.067862 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.290888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.270407 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:04:15.337382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.318861 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.337382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.319306 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:04:15.337382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.319845 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.511066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.509890 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:04:15.696074 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:15.692617 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.24s" Nov 01 16:04:15.789565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.789536 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:04:15.791783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.789914 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:04:15.820903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.820099 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:15.853277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.853238 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" event={"ID":"4975fc93-d65c-436c-a92e-035e40e2aaa3","Type":"ContainerStarted","Data":"6a6fcc99adbdff4bd05fcb9026f7346ce0b98505606c1e890d01c3e10e35acb4"} Nov 01 16:04:16.003717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:15.952937 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.069061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.069009 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.104775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.104326 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.104980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.104865 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 01 16:04:16.283829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.281531 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.328781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.328055 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.328781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.328414 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:04:16.328781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.328656 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:04:16.668830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.512277 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.717093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.711700 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:04:16.717093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.711784 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:16.926828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:16.925866 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.092237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.092187 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.093378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.093324 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.095277 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:17.095247 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:04:17.137312 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:17.137264 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:04:17.243471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.243363 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:04:17.339777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.307647 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded" Nov 01 16:04:17.429778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.334167 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.601397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.336256 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:04:17.628781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.336291 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:04:17.630374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.513036 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:17.639782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.551232 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:04:17.833775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.562160 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt"] Nov 01 16:04:17.888087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.888054 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:04:17.953404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:17.953363 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.095175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.095127 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.127236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.118433 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.585959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.585893 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.618865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.618813 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.673781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.644613 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:04:18.673781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.644991 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:04:18.784800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.766740 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.934464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.787427 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:18.934842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.820892 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-mrk7c" podUID="eedaa33d-b796-42e7-8a32-403dc27c0221" containerName="main" probeResult="failure" output="Get \"https://10.133.0.35:8000/ready\": context deadline exceeded" Nov 01 16:04:18.936556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.820989 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-jnxjv" podUID="2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.34:8000/ready\": context deadline exceeded" Nov 01 16:04:18.938073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.933006 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:18.987652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:18.985862 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.060883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.060789 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:04:19.134771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.127881 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.231778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.197966 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:04:19.354868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.333045 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Nov 01 16:04:19.378358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.378327 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:04:19.420622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.409583 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" event={"ID":"4975fc93-d65c-436c-a92e-035e40e2aaa3","Type":"ContainerStarted","Data":"45f87fed7ebe19bea4624e1b501155466184d8a3b47bd498bdeaa0e2af60aab3"} Nov 01 16:04:19.477207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.477176 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:04:19.497828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.487989 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:04:19.497828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.491385 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:04:19.497828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.492495 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:04:19.601856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.600486 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:04:19.607360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.607340 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:04:19.644534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.644468 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:04:19.680108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.680054 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" podStartSLOduration=8.680034809 podStartE2EDuration="8.680034809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:19.433549888 +0000 UTC m=+997.136715826" watchObservedRunningTime="2026-11-01 16:04:19.680034809 +0000 UTC m=+997.383200747" Nov 01 16:04:19.902005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:19.901892 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:20.421403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:20.421361 4573 generic.go:356] "Generic (PLEG): container finished" podID="4975fc93-d65c-436c-a92e-035e40e2aaa3" containerID="45f87fed7ebe19bea4624e1b501155466184d8a3b47bd498bdeaa0e2af60aab3" exitCode=0 Nov 01 16:04:20.421986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:20.421460 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" event={"ID":"4975fc93-d65c-436c-a92e-035e40e2aaa3","Type":"ContainerDied","Data":"45f87fed7ebe19bea4624e1b501155466184d8a3b47bd498bdeaa0e2af60aab3"} Nov 01 16:04:21.095863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.095827 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:21.615220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.615194 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:21.719532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.719450 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-openshift-ca\") pod \"4975fc93-d65c-436c-a92e-035e40e2aaa3\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " Nov 01 16:04:21.719532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.719519 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4975fc93-d65c-436c-a92e-035e40e2aaa3-kube-api-access-hn462\" (UniqueName: \"kubernetes.io/projected/4975fc93-d65c-436c-a92e-035e40e2aaa3-kube-api-access-hn462\") pod \"4975fc93-d65c-436c-a92e-035e40e2aaa3\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " Nov 01 16:04:21.719790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.719660 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-xks-ca\") pod \"4975fc93-d65c-436c-a92e-035e40e2aaa3\" (UID: \"4975fc93-d65c-436c-a92e-035e40e2aaa3\") " Nov 01 16:04:21.719911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.719885 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-openshift-ca" pod "4975fc93-d65c-436c-a92e-035e40e2aaa3" (UID: "4975fc93-d65c-436c-a92e-035e40e2aaa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:21.720492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.720466 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:21.721655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.721635 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4975fc93-d65c-436c-a92e-035e40e2aaa3-kube-api-access-hn462" pod "4975fc93-d65c-436c-a92e-035e40e2aaa3" (UID: "4975fc93-d65c-436c-a92e-035e40e2aaa3"). InnerVolumeSpecName "kube-api-access-hn462". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:21.721949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.721931 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-xks-ca" pod "4975fc93-d65c-436c-a92e-035e40e2aaa3" (UID: "4975fc93-d65c-436c-a92e-035e40e2aaa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:21.821702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.821666 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn462\" (UniqueName: \"kubernetes.io/projected/4975fc93-d65c-436c-a92e-035e40e2aaa3-kube-api-access-hn462\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:21.821702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.821695 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4975fc93-d65c-436c-a92e-035e40e2aaa3-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:21.986902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:21.986729 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:22.443598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.443570 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" Nov 01 16:04:22.443817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.443603 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt" event={"ID":"4975fc93-d65c-436c-a92e-035e40e2aaa3","Type":"ContainerDied","Data":"6a6fcc99adbdff4bd05fcb9026f7346ce0b98505606c1e890d01c3e10e35acb4"} Nov 01 16:04:22.443817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.443695 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a6fcc99adbdff4bd05fcb9026f7346ce0b98505606c1e890d01c3e10e35acb4" Nov 01 16:04:22.746322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.746232 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:22.916659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.916624 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx"] Nov 01 16:04:22.917455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.917431 4573 cpu_manager.go:414] "containerMap: removing container" podUID="4975fc93-d65c-436c-a92e-035e40e2aaa3" containerName="revoke-keys" Nov 01 16:04:22.917455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.917452 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4975fc93-d65c-436c-a92e-035e40e2aaa3" containerName="revoke-keys" Nov 01 16:04:22.917629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.917579 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="4975fc93-d65c-436c-a92e-035e40e2aaa3" containerName="revoke-keys" Nov 01 16:04:22.923068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.923046 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:22.929706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.929678 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx"] Nov 01 16:04:22.930611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.930577 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:04:22.930742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:22.930690 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:04:23.051080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.050991 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.051361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.051341 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.051424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.051382 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xj8\" (UniqueName: \"kubernetes.io/projected/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-kube-api-access-k2xj8\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.153012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.152964 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.153012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.153026 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xj8\" (UniqueName: \"kubernetes.io/projected/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-kube-api-access-k2xj8\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.153291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.153061 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.153782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.153731 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.155382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.155363 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.160727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.160704 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xj8\" (UniqueName: \"kubernetes.io/projected/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-kube-api-access-k2xj8\") pod \"maas-api-revoke-keys-e2e-migrate-3664624a-75pkx\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.244805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.244774 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:23.533370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.533347 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx"] Nov 01 16:04:23.535939 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:04:23.535911 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6172e6cc_d1d3_4726_ae6f_fe5d6df84bbb.slice/crio-40d61137b7432c2f1c7013fe307b79e03f4462c1a8685857e2de4daa829466a3 WatchSource:0}: Error finding container 40d61137b7432c2f1c7013fe307b79e03f4462c1a8685857e2de4daa829466a3: Status 404 returned error can't find the container with id 40d61137b7432c2f1c7013fe307b79e03f4462c1a8685857e2de4daa829466a3 Nov 01 16:04:23.769943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:23.769909 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:24.365266 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:24.365232 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:04:24.470793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:24.467556 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" event={"ID":"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb","Type":"ContainerStarted","Data":"c2143545197cc383f3b11aa08fa63b40f996ea65f6dad89034f1c4f679e89253"} Nov 01 16:04:24.470793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:24.467624 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" event={"ID":"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb","Type":"ContainerStarted","Data":"40d61137b7432c2f1c7013fe307b79e03f4462c1a8685857e2de4daa829466a3"} Nov 01 16:04:24.473966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:24.473935 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:24.489997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:24.489923 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" podStartSLOduration=2.489904619 podStartE2EDuration="2.489904619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:24.483062894 +0000 UTC m=+1002.186228840" watchObservedRunningTime="2026-11-01 16:04:24.489904619 +0000 UTC m=+1002.193070556" Nov 01 16:04:25.482006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:25.481946 4573 generic.go:356] "Generic (PLEG): container finished" podID="6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb" containerID="c2143545197cc383f3b11aa08fa63b40f996ea65f6dad89034f1c4f679e89253" exitCode=0 Nov 01 16:04:25.482525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:25.482013 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" event={"ID":"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb","Type":"ContainerDied","Data":"c2143545197cc383f3b11aa08fa63b40f996ea65f6dad89034f1c4f679e89253"} Nov 01 16:04:25.517438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:25.517403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:26.263521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.263487 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:26.666063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.666039 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:26.740871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.740835 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-openshift-ca\") pod \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " Nov 01 16:04:26.741067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.741045 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-xks-ca\") pod \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " Nov 01 16:04:26.741149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.741093 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-kube-api-access-k2xj8\" (UniqueName: \"kubernetes.io/projected/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-kube-api-access-k2xj8\") pod \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\" (UID: \"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb\") " Nov 01 16:04:26.741233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.741198 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-openshift-ca" pod "6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb" (UID: "6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:26.741780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.741743 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:26.743403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.743370 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-xks-ca" pod "6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb" (UID: "6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:26.743650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.743629 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-kube-api-access-k2xj8" pod "6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb" (UID: "6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb"). InnerVolumeSpecName "kube-api-access-k2xj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:26.842952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.842879 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:26.842952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:26.842906 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2xj8\" (UniqueName: \"kubernetes.io/projected/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb-kube-api-access-k2xj8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:27.056850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.056818 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:27.501335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.501307 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" Nov 01 16:04:27.501540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.501336 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx" event={"ID":"6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb","Type":"ContainerDied","Data":"40d61137b7432c2f1c7013fe307b79e03f4462c1a8685857e2de4daa829466a3"} Nov 01 16:04:27.501540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.501388 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40d61137b7432c2f1c7013fe307b79e03f4462c1a8685857e2de4daa829466a3" Nov 01 16:04:27.584925 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:27.584892 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:04:27.777717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.777636 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k"] Nov 01 16:04:27.778797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.778773 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb" containerName="revoke-keys" Nov 01 16:04:27.778871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.778799 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb" containerName="revoke-keys" Nov 01 16:04:27.779097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.779081 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb" containerName="revoke-keys" Nov 01 16:04:27.783940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.783919 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.798686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.798658 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k"] Nov 01 16:04:27.833404 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:27.833369 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:04:27.860716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.860679 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.860920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.860729 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.860920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.860847 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.860920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.860898 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.861073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.861049 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.861200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.861178 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.861322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.861306 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66pbz\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-kube-api-access-66pbz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.947369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.947341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:27.963040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.963012 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.963189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.963060 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.963189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.963102 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.963189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.963151 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.963369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.963289 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.963425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.963413 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.963552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.963524 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66pbz\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-kube-api-access-66pbz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.964041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.964020 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.965480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.965455 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.965653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.965635 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.965864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.965846 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.966013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.965992 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.970904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.970882 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66pbz\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-kube-api-access-66pbz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:27.971072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:27.971053 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rtv2k\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:28.106275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.106188 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:28.414102 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:04:28.414068 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d266dd_6807_4e9c_bb5e_59fb4d3c2194.slice/crio-fb2f232fdd32ea83fe2702ba738dabe89be41d2a8a3e6705bffd07dad08a2634 WatchSource:0}: Error finding container fb2f232fdd32ea83fe2702ba738dabe89be41d2a8a3e6705bffd07dad08a2634: Status 404 returned error can't find the container with id fb2f232fdd32ea83fe2702ba738dabe89be41d2a8a3e6705bffd07dad08a2634 Nov 01 16:04:28.418106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.417957 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k"] Nov 01 16:04:28.418480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.418434 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:28.418601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.418577 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:04:28.514536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.513319 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" event={"ID":"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194","Type":"ContainerStarted","Data":"295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921"} Nov 01 16:04:28.514536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.513390 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" event={"ID":"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194","Type":"ContainerStarted","Data":"fb2f232fdd32ea83fe2702ba738dabe89be41d2a8a3e6705bffd07dad08a2634"} Nov 01 16:04:28.514536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.514488 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:28.536568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.536498 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" podStartSLOduration=1.536480803 podStartE2EDuration="1.536480803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:28.535669266 +0000 UTC m=+1006.238835203" watchObservedRunningTime="2026-11-01 16:04:28.536480803 +0000 UTC m=+1006.239646739" Nov 01 16:04:28.939554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:28.939518 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:29.522027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.521989 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m"] Nov 01 16:04:29.528628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.528326 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.533818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.532732 4573 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rtv2k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:04:29.533818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.532801 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" podUID="e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:04:29.533818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.533636 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:04:29.533818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.533765 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:04:29.539393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.539361 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m"] Nov 01 16:04:29.635123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.635087 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4bfc\" (UniqueName: \"kubernetes.io/projected/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-kube-api-access-n4bfc\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.635513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.635491 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.635709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.635683 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.737106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.737065 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.737257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.737198 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4bfc\" (UniqueName: \"kubernetes.io/projected/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-kube-api-access-n4bfc\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.737346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.737333 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.737814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.737786 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.739688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.739662 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.745700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.745670 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4bfc\" (UniqueName: \"kubernetes.io/projected/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-kube-api-access-n4bfc\") pod \"maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.863962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.863930 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:29.957046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:29.957007 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:30.154819 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:04:30.151834 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb096c055_e37f_4cdb_8063_ea0e3a5f6fc3.slice/crio-300b02408da5924d3bf56eee0c5c919bf5ae1b8442a0c6d812dbef23f23bf488 WatchSource:0}: Error finding container 300b02408da5924d3bf56eee0c5c919bf5ae1b8442a0c6d812dbef23f23bf488: Status 404 returned error can't find the container with id 300b02408da5924d3bf56eee0c5c919bf5ae1b8442a0c6d812dbef23f23bf488 Nov 01 16:04:30.155104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:30.155083 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m"] Nov 01 16:04:30.539048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:30.539013 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" event={"ID":"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3","Type":"ContainerStarted","Data":"d8f4410c99f1dff79940db2710a10bbf42023e115746fba5d98c3c21c4c0fa74"} Nov 01 16:04:30.539202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:30.539071 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" event={"ID":"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3","Type":"ContainerStarted","Data":"300b02408da5924d3bf56eee0c5c919bf5ae1b8442a0c6d812dbef23f23bf488"} Nov 01 16:04:30.539765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:30.539729 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:04:30.558102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:30.558060 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" podStartSLOduration=1.558047921 podStartE2EDuration="1.558047921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:04:30.553202998 +0000 UTC m=+1008.256368934" watchObservedRunningTime="2026-11-01 16:04:30.558047921 +0000 UTC m=+1008.261213857" Nov 01 16:04:30.715053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:30.715024 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:31.049518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:31.049485 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:31.370536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:31.370499 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s"] Nov 01 16:04:31.432915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:31.432879 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:31.534529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:31.534491 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:31.551269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:31.551235 4573 generic.go:356] "Generic (PLEG): container finished" podID="b096c055-e37f-4cdb-8063-ea0e3a5f6fc3" containerID="d8f4410c99f1dff79940db2710a10bbf42023e115746fba5d98c3c21c4c0fa74" exitCode=0 Nov 01 16:04:31.551404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:31.551316 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" event={"ID":"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3","Type":"ContainerDied","Data":"d8f4410c99f1dff79940db2710a10bbf42023e115746fba5d98c3c21c4c0fa74"} Nov 01 16:04:31.629897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:31.629788 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:31.975835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:31.975657 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:32.508612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.508581 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:32.601526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.601491 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:32.730058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.730035 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:32.873630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.873601 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-openshift-ca\") pod \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " Nov 01 16:04:32.873842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.873722 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-xks-ca\") pod \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " Nov 01 16:04:32.874003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.873907 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-kube-api-access-n4bfc\" (UniqueName: \"kubernetes.io/projected/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-kube-api-access-n4bfc\") pod \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\" (UID: \"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3\") " Nov 01 16:04:32.874266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.874237 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-openshift-ca" pod "b096c055-e37f-4cdb-8063-ea0e3a5f6fc3" (UID: "b096c055-e37f-4cdb-8063-ea0e3a5f6fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:32.874782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.874701 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:32.876236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.876214 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-kube-api-access-n4bfc" pod "b096c055-e37f-4cdb-8063-ea0e3a5f6fc3" (UID: "b096c055-e37f-4cdb-8063-ea0e3a5f6fc3"). InnerVolumeSpecName "kube-api-access-n4bfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:32.876445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.876423 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-xks-ca" pod "b096c055-e37f-4cdb-8063-ea0e3a5f6fc3" (UID: "b096c055-e37f-4cdb-8063-ea0e3a5f6fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:04:32.941294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.941245 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:32.976559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.976524 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4bfc\" (UniqueName: \"kubernetes.io/projected/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-kube-api-access-n4bfc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:32.976559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:32.976561 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:33.054774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:33.052918 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:33.152903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:33.152820 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:33.227195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:33.227162 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:33.584599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:33.584528 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" Nov 01 16:04:33.584982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:33.584530 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m" event={"ID":"b096c055-e37f-4cdb-8063-ea0e3a5f6fc3","Type":"ContainerDied","Data":"300b02408da5924d3bf56eee0c5c919bf5ae1b8442a0c6d812dbef23f23bf488"} Nov 01 16:04:33.584982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:33.584648 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="300b02408da5924d3bf56eee0c5c919bf5ae1b8442a0c6d812dbef23f23bf488" Nov 01 16:04:33.865788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:33.865741 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:34.565573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:34.565524 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:35.211862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:35.211831 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:35.873318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:35.873286 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:36.371869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.371803 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" podUID="a757c981-efd8-45bb-8b5b-b1754ed1c6ad" containerName="payload-processing" containerID="cri-o://295185b6e8b31d1f5d16f66f0db9b9bc8afd49c68bb13e8f140e5d76d4144b49" gracePeriod=25 Nov 01 16:04:36.502434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.502403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:36.615187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.615144 4573 generic.go:356] "Generic (PLEG): container finished" podID="a757c981-efd8-45bb-8b5b-b1754ed1c6ad" containerID="295185b6e8b31d1f5d16f66f0db9b9bc8afd49c68bb13e8f140e5d76d4144b49" exitCode=0 Nov 01 16:04:36.615370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.615210 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" event={"ID":"a757c981-efd8-45bb-8b5b-b1754ed1c6ad","Type":"ContainerDied","Data":"295185b6e8b31d1f5d16f66f0db9b9bc8afd49c68bb13e8f140e5d76d4144b49"} Nov 01 16:04:36.634221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.634136 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:36.792881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.785603 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:36.879985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.879954 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:04:36.960969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.949578 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-plugins-config-volume\") pod \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\" (UID: \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\") " Nov 01 16:04:36.960969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.950039 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-kube-api-access-6k4vv\" (UniqueName: \"kubernetes.io/projected/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-kube-api-access-6k4vv\") pod \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\" (UID: \"a757c981-efd8-45bb-8b5b-b1754ed1c6ad\") " Nov 01 16:04:36.960969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.951691 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-plugins-config-volume" pod "a757c981-efd8-45bb-8b5b-b1754ed1c6ad" (UID: "a757c981-efd8-45bb-8b5b-b1754ed1c6ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:36.968642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.968611 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt"] Nov 01 16:04:36.971315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.971281 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-kube-api-access-6k4vv" pod "a757c981-efd8-45bb-8b5b-b1754ed1c6ad" (UID: "a757c981-efd8-45bb-8b5b-b1754ed1c6ad"). InnerVolumeSpecName "kube-api-access-6k4vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:36.985583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:36.985542 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8624bae9-47slt"] Nov 01 16:04:37.033108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.033073 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:37.040267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.040233 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq"] Nov 01 16:04:37.040520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.040488 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" podUID="1770918f-862c-4189-9eef-1260c25f619d" containerName="istio-proxy" containerID="cri-o://fbeeee9817eaad016ee7637ef57c041ab0072df35bdb05fe95ca28e1f76d5eba" gracePeriod=30 Nov 01 16:04:37.054517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.054486 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k4vv\" (UniqueName: \"kubernetes.io/projected/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-kube-api-access-6k4vv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:37.054517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.054516 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a757c981-efd8-45bb-8b5b-b1754ed1c6ad-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:37.190795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.186195 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:37.626012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.625981 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" Nov 01 16:04:37.626012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.626005 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s" event={"ID":"a757c981-efd8-45bb-8b5b-b1754ed1c6ad","Type":"ContainerDied","Data":"2b133b55a5e236b4057a92b17a112bd8e0ce62b55171e56581a8d628a9b98008"} Nov 01 16:04:37.626518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.626071 4573 scope.go:122] "RemoveContainer" containerID="295185b6e8b31d1f5d16f66f0db9b9bc8afd49c68bb13e8f140e5d76d4144b49" Nov 01 16:04:37.653124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.653073 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s"] Nov 01 16:04:37.659402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.659374 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8624bae9-548c6d49df-p5h5s"] Nov 01 16:04:37.720977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:37.720941 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:37.943929 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:37.943820 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:04:38.065643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:38.065610 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:38.151331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:38.151283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:38.476080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:38.476040 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4975fc93-d65c-436c-a92e-035e40e2aaa3" path="/var/lib/kubelet/pods/4975fc93-d65c-436c-a92e-035e40e2aaa3/volumes" Nov 01 16:04:38.476581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:38.476566 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a757c981-efd8-45bb-8b5b-b1754ed1c6ad" path="/var/lib/kubelet/pods/a757c981-efd8-45bb-8b5b-b1754ed1c6ad/volumes" Nov 01 16:04:38.889519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:38.889485 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:39.263119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:39.263033 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:39.559537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:39.559339 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:39.727396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:39.727354 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:39.819511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:39.819419 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:39.948775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:39.943894 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:39.983832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:39.980889 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx"] Nov 01 16:04:40.037930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:40.037886 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3664624a-75pkx"] Nov 01 16:04:40.064771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:40.063432 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:40.213356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:40.213258 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:40.462840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:40.462799 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb" path="/var/lib/kubelet/pods/6172e6cc-d1d3-4726-ae6f-fe5d6df84bbb/volumes" Nov 01 16:04:40.808918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:40.808883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:41.548702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:41.548642 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:41.880776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:41.878197 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:42.031066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.031020 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:42.567770 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:42.566610 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-conmon-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227419a0_7ea0_40df_a1c1_68228db79e84.slice/crio-44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:04:42.818924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.818846 4573 generic.go:356] "Generic (PLEG): container finished" podID="1770918f-862c-4189-9eef-1260c25f619d" containerID="fbeeee9817eaad016ee7637ef57c041ab0072df35bdb05fe95ca28e1f76d5eba" exitCode=0 Nov 01 16:04:42.819114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.819088 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" event={"ID":"1770918f-862c-4189-9eef-1260c25f619d","Type":"ContainerDied","Data":"fbeeee9817eaad016ee7637ef57c041ab0072df35bdb05fe95ca28e1f76d5eba"} Nov 01 16:04:42.821475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.821423 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:04:42.900665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.900439 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-kube-api-access-pc2b5\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-kube-api-access-pc2b5\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.900665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.900558 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-istio-token\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-istio-token\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.900665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.900612 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1770918f-862c-4189-9eef-1260c25f619d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1770918f-862c-4189-9eef-1260c25f619d-istiod-ca-cert\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.901019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.900675 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-envoy\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.901019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.900735 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-socket\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.901123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.901020 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-data\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.901173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.901150 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1770918f-862c-4189-9eef-1260c25f619d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1770918f-862c-4189-9eef-1260c25f619d-istio-podinfo\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.901256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.901228 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-credential-socket\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.901321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.901286 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-certs\") pod \"1770918f-862c-4189-9eef-1260c25f619d\" (UID: \"1770918f-862c-4189-9eef-1260c25f619d\") " Nov 01 16:04:42.908100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.905222 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-certs" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:42.908100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.905467 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-socket" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:42.908100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.905631 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-data" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:42.908100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.907429 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1770918f-862c-4189-9eef-1260c25f619d-istiod-ca-cert" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:42.908100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.907624 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-envoy" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:42.909104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.909078 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-credential-socket" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:42.909927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.909896 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1770918f-862c-4189-9eef-1260c25f619d-istio-podinfo" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:04:42.911093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.911011 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:42.911414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.911386 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-istio-token" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:42.919514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:42.919486 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-kube-api-access-pc2b5" pod "1770918f-862c-4189-9eef-1260c25f619d" (UID: "1770918f-862c-4189-9eef-1260c25f619d"). InnerVolumeSpecName "kube-api-access-pc2b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:43.002635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.002586 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004134 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004163 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1770918f-862c-4189-9eef-1260c25f619d-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004181 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004194 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004208 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pc2b5\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-kube-api-access-pc2b5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004223 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1770918f-862c-4189-9eef-1260c25f619d-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004235 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1770918f-862c-4189-9eef-1260c25f619d-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004248 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.004508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.004262 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1770918f-862c-4189-9eef-1260c25f619d-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:43.787842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.784161 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:43.897779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.895659 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" event={"ID":"1770918f-862c-4189-9eef-1260c25f619d","Type":"ContainerDied","Data":"2520de558a5fd39b23fcb231a50b7c5ffa3c38769b1bcd3cf1f147dd5861dae7"} Nov 01 16:04:43.897779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.895736 4573 scope.go:122] "RemoveContainer" containerID="fbeeee9817eaad016ee7637ef57c041ab0072df35bdb05fe95ca28e1f76d5eba" Nov 01 16:04:43.897779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.895946 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq" Nov 01 16:04:43.935192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.935140 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq"] Nov 01 16:04:43.943932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:43.943897 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-8624bae9-openshift-default-5b6bc89bc5-r29nq"] Nov 01 16:04:44.460037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:44.460005 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1770918f-862c-4189-9eef-1260c25f619d" path="/var/lib/kubelet/pods/1770918f-862c-4189-9eef-1260c25f619d/volumes" Nov 01 16:04:44.660560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:44.660525 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:45.397892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:45.397859 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:46.138121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:46.138087 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:46.466828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:46.449372 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:46.737223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:46.737090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:46.824775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:46.823798 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m"] Nov 01 16:04:46.838901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:46.838575 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec6894c9-4b95m"] Nov 01 16:04:46.906893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:46.904486 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:46.995223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:46.995136 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:47.426318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:47.426279 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm"] Nov 01 16:04:47.426598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:47.426544 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" podUID="221b3e5d-7d63-468e-ae3a-0f9aec607585" containerName="istio-proxy" containerID="cri-o://83e5ade1142854dd688ed9806c224f2a13436a62075b90dba5d512b9bd7dac50" gracePeriod=30 Nov 01 16:04:47.430125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:47.429156 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:48.096230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:48.096118 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:48.182964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:48.182934 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:48.284096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:48.284052 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:48.381053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:48.381020 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:48.598774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:48.591730 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b096c055-e37f-4cdb-8063-ea0e3a5f6fc3" path="/var/lib/kubelet/pods/b096c055-e37f-4cdb-8063-ea0e3a5f6fc3/volumes" Nov 01 16:04:49.245065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:49.245019 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:50.010563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:50.010520 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:50.753381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:50.753342 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:51.481034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:51.480991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:51.617776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:51.617352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:51.771208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:51.771090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:52.460927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:52.460881 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:52.999971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:52.999885 4573 generic.go:356] "Generic (PLEG): container finished" podID="221b3e5d-7d63-468e-ae3a-0f9aec607585" containerID="83e5ade1142854dd688ed9806c224f2a13436a62075b90dba5d512b9bd7dac50" exitCode=0 Nov 01 16:04:52.999971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:52.999954 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" event={"ID":"221b3e5d-7d63-468e-ae3a-0f9aec607585","Type":"ContainerDied","Data":"83e5ade1142854dd688ed9806c224f2a13436a62075b90dba5d512b9bd7dac50"} Nov 01 16:04:53.139909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.139884 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:04:53.145090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.145068 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:53.278479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.278439 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-token\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-token\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.278697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.278583 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-podinfo\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.278697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.278642 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-data\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.278897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.278712 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-envoy\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.278897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.278775 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-certs\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.278897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.278872 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/221b3e5d-7d63-468e-ae3a-0f9aec607585-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b3e5d-7d63-468e-ae3a-0f9aec607585-istiod-ca-cert\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.279210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.279050 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-kube-api-access-fzqz9\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-kube-api-access-fzqz9\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.279210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.279098 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-socket\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.279210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.279147 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-credential-socket\") pod \"221b3e5d-7d63-468e-ae3a-0f9aec607585\" (UID: \"221b3e5d-7d63-468e-ae3a-0f9aec607585\") " Nov 01 16:04:53.280306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.280284 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-certs" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:53.280803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.280465 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-socket" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:53.280919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.280482 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-data" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:53.280919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.280660 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/221b3e5d-7d63-468e-ae3a-0f9aec607585-istiod-ca-cert" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:04:53.280919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.280788 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-credential-socket" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:53.283613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.283590 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-envoy" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:04:53.284366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.284334 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-token" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:53.285455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.285431 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-kube-api-access-fzqz9" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "kube-api-access-fzqz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:04:53.294997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.294953 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-podinfo" pod "221b3e5d-7d63-468e-ae3a-0f9aec607585" (UID: "221b3e5d-7d63-468e-ae3a-0f9aec607585"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382561 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382590 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/221b3e5d-7d63-468e-ae3a-0f9aec607585-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382600 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzqz9\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-kube-api-access-fzqz9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382611 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382619 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382628 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382636 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382644 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.382646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.382653 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/221b3e5d-7d63-468e-ae3a-0f9aec607585-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:04:53.867083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:53.867047 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:54.184192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:54.184098 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" event={"ID":"221b3e5d-7d63-468e-ae3a-0f9aec607585","Type":"ContainerDied","Data":"ac797f514f1f00769e80ca82ee7f9de95623af842fd591d9dd4be98a316f64b9"} Nov 01 16:04:54.184192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:54.184187 4573 scope.go:122] "RemoveContainer" containerID="83e5ade1142854dd688ed9806c224f2a13436a62075b90dba5d512b9bd7dac50" Nov 01 16:04:54.184734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:54.184445 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm" Nov 01 16:04:54.219769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:54.219715 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm"] Nov 01 16:04:54.226975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:54.226950 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ec6894c9-openshift-default-6f8c69ffc9-sjgjm"] Nov 01 16:04:54.365812 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:04:54.365775 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:04:54.460301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:54.460221 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="221b3e5d-7d63-468e-ae3a-0f9aec607585" path="/var/lib/kubelet/pods/221b3e5d-7d63-468e-ae3a-0f9aec607585/volumes" Nov 01 16:04:54.661212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:54.661175 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:55.553413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:55.553371 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:55.653769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:55.652511 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:55.788167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:55.788134 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:56.661632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:56.661593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:57.395689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:57.395630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:58.177463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:58.177423 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:58.987442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:58.987407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:04:59.874146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:04:59.874097 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:00.668936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:00.668902 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:01.407685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:01.407644 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:02.469335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:02.469292 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:03.134897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:03.134859 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:03.783499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:03.783466 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:04.475184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:04.475151 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:04.616307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:04.616270 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:05.362938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:05.362902 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:06.070603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.070571 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:06.668336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.668301 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:06.696069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.696032 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb"] Nov 01 16:05:06.697203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697178 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1770918f-862c-4189-9eef-1260c25f619d" containerName="istio-proxy" Nov 01 16:05:06.697203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697202 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1770918f-862c-4189-9eef-1260c25f619d" containerName="istio-proxy" Nov 01 16:05:06.697362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697219 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b096c055-e37f-4cdb-8063-ea0e3a5f6fc3" containerName="revoke-keys" Nov 01 16:05:06.697362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697227 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b096c055-e37f-4cdb-8063-ea0e3a5f6fc3" containerName="revoke-keys" Nov 01 16:05:06.697362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697266 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a757c981-efd8-45bb-8b5b-b1754ed1c6ad" containerName="payload-processing" Nov 01 16:05:06.697362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697275 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a757c981-efd8-45bb-8b5b-b1754ed1c6ad" containerName="payload-processing" Nov 01 16:05:06.697362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697288 4573 cpu_manager.go:414] "containerMap: removing container" podUID="221b3e5d-7d63-468e-ae3a-0f9aec607585" containerName="istio-proxy" Nov 01 16:05:06.697362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697295 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="221b3e5d-7d63-468e-ae3a-0f9aec607585" containerName="istio-proxy" Nov 01 16:05:06.697544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697496 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1770918f-862c-4189-9eef-1260c25f619d" containerName="istio-proxy" Nov 01 16:05:06.697544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697511 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b096c055-e37f-4cdb-8063-ea0e3a5f6fc3" containerName="revoke-keys" Nov 01 16:05:06.697544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697526 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a757c981-efd8-45bb-8b5b-b1754ed1c6ad" containerName="payload-processing" Nov 01 16:05:06.697544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.697534 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="221b3e5d-7d63-468e-ae3a-0f9aec607585" containerName="istio-proxy" Nov 01 16:05:06.700806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.700785 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:06.709013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.708985 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:05:06.709242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.709222 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:05:06.710263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.710242 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb"] Nov 01 16:05:06.940158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.940061 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8nmz\" (UniqueName: \"kubernetes.io/projected/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-kube-api-access-d8nmz\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:06.940158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.940146 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:06.940359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:06.940309 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:07.041672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.041629 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8nmz\" (UniqueName: \"kubernetes.io/projected/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-kube-api-access-d8nmz\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:07.041885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.041697 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:07.041885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.041831 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:07.042420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.042391 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:07.044237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.044215 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:07.049598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.049566 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8nmz\" (UniqueName: \"kubernetes.io/projected/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-kube-api-access-d8nmz\") pod \"maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:07.333650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.333569 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:07.469168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.469095 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:07.604179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.604086 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh"] Nov 01 16:05:07.614975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.614930 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh"] Nov 01 16:05:07.624855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.624822 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.646494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.643630 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb"] Nov 01 16:05:07.659872 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:05:07.659843 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bd5a5c_dc2c_42ed_92a2_c1a0eb90507a.slice/crio-e0c5cbfc8921b88076a021c7893eb73a4f6b26b7429b89f5d231c2dc460893b6 WatchSource:0}: Error finding container e0c5cbfc8921b88076a021c7893eb73a4f6b26b7429b89f5d231c2dc460893b6: Status 404 returned error can't find the container with id e0c5cbfc8921b88076a021c7893eb73a4f6b26b7429b89f5d231c2dc460893b6 Nov 01 16:05:07.661952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.661916 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfc6p\" (UniqueName: \"kubernetes.io/projected/ec731ac5-49a1-4667-887a-b500f3732f2f-kube-api-access-mfc6p\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.662100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.661997 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.662206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.662190 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.763688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.763657 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfc6p\" (UniqueName: \"kubernetes.io/projected/ec731ac5-49a1-4667-887a-b500f3732f2f-kube-api-access-mfc6p\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.764176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.763741 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.764176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.763954 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.764471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.764448 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.766991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.766967 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.771073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.771054 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfc6p\" (UniqueName: \"kubernetes.io/projected/ec731ac5-49a1-4667-887a-b500f3732f2f-kube-api-access-mfc6p\") pod \"maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:07.960070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:07.960037 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:08.246693 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:05:08.246651 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec731ac5_49a1_4667_887a_b500f3732f2f.slice/crio-b88eb50166a74393fe278d80acde56a48f25bf91c94c5fc196be24233a9ae77a WatchSource:0}: Error finding container b88eb50166a74393fe278d80acde56a48f25bf91c94c5fc196be24233a9ae77a: Status 404 returned error can't find the container with id b88eb50166a74393fe278d80acde56a48f25bf91c94c5fc196be24233a9ae77a Nov 01 16:05:08.249354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:08.249326 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh"] Nov 01 16:05:08.426103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:08.426064 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:08.426281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:08.426070 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" event={"ID":"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a","Type":"ContainerStarted","Data":"b876968002526433e91e84d15ed57a570ca9e45e75769cdd2823fa8f1bd961ae"} Nov 01 16:05:08.426281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:08.426168 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" event={"ID":"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a","Type":"ContainerStarted","Data":"e0c5cbfc8921b88076a021c7893eb73a4f6b26b7429b89f5d231c2dc460893b6"} Nov 01 16:05:08.427799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:08.427773 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" event={"ID":"ec731ac5-49a1-4667-887a-b500f3732f2f","Type":"ContainerStarted","Data":"b88eb50166a74393fe278d80acde56a48f25bf91c94c5fc196be24233a9ae77a"} Nov 01 16:05:08.446398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:08.446333 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" podStartSLOduration=2.446315712 podStartE2EDuration="2.446315712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:08.440259827 +0000 UTC m=+1046.143425765" watchObservedRunningTime="2026-11-01 16:05:08.446315712 +0000 UTC m=+1046.149481647" Nov 01 16:05:09.164138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:09.164100 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:09.439013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:09.438926 4573 generic.go:356] "Generic (PLEG): container finished" podID="61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a" containerID="b876968002526433e91e84d15ed57a570ca9e45e75769cdd2823fa8f1bd961ae" exitCode=0 Nov 01 16:05:09.439187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:09.439013 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" event={"ID":"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a","Type":"ContainerDied","Data":"b876968002526433e91e84d15ed57a570ca9e45e75769cdd2823fa8f1bd961ae"} Nov 01 16:05:09.440763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:09.440726 4573 generic.go:356] "Generic (PLEG): container finished" podID="ec731ac5-49a1-4667-887a-b500f3732f2f" containerID="98d025663e64a875f77297aab0705c34904035cd1937852b2e480eaf12f0b289" exitCode=0 Nov 01 16:05:09.440895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:09.440810 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" event={"ID":"ec731ac5-49a1-4667-887a-b500f3732f2f","Type":"ContainerDied","Data":"98d025663e64a875f77297aab0705c34904035cd1937852b2e480eaf12f0b289"} Nov 01 16:05:09.815369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:09.815282 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:10.084862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.081156 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl"] Nov 01 16:05:10.110776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.110018 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea7d7981-jjkfl"] Nov 01 16:05:10.459076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.459033 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="621564cb-729f-41dc-986a-865e7928511c" path="/var/lib/kubelet/pods/621564cb-729f-41dc-986a-865e7928511c/volumes" Nov 01 16:05:10.588316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.588286 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:10.660911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.660881 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:10.664927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.664906 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:10.681903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.681872 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:10.726646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.726555 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-kube-api-access-d8nmz\" (UniqueName: \"kubernetes.io/projected/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-kube-api-access-d8nmz\") pod \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " Nov 01 16:05:10.726646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.726610 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-xks-ca\") pod \"ec731ac5-49a1-4667-887a-b500f3732f2f\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " Nov 01 16:05:10.726897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.726692 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-openshift-ca\") pod \"ec731ac5-49a1-4667-887a-b500f3732f2f\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " Nov 01 16:05:10.726897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.726735 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-xks-ca\") pod \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " Nov 01 16:05:10.726897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.726819 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-openshift-ca\") pod \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\" (UID: \"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a\") " Nov 01 16:05:10.727088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.726915 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec731ac5-49a1-4667-887a-b500f3732f2f-kube-api-access-mfc6p\" (UniqueName: \"kubernetes.io/projected/ec731ac5-49a1-4667-887a-b500f3732f2f-kube-api-access-mfc6p\") pod \"ec731ac5-49a1-4667-887a-b500f3732f2f\" (UID: \"ec731ac5-49a1-4667-887a-b500f3732f2f\") " Nov 01 16:05:10.727143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.727067 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-openshift-ca" pod "ec731ac5-49a1-4667-887a-b500f3732f2f" (UID: "ec731ac5-49a1-4667-887a-b500f3732f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:10.727253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.727229 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-openshift-ca" pod "61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a" (UID: "61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:05:10.728059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.728028 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:10.728195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.728065 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:10.729306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.729278 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-kube-api-access-d8nmz" pod "61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a" (UID: "61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a"). InnerVolumeSpecName "kube-api-access-d8nmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:10.729709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.729683 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-xks-ca" pod "ec731ac5-49a1-4667-887a-b500f3732f2f" (UID: "ec731ac5-49a1-4667-887a-b500f3732f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:05:10.730133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.730098 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec731ac5-49a1-4667-887a-b500f3732f2f-kube-api-access-mfc6p" pod "ec731ac5-49a1-4667-887a-b500f3732f2f" (UID: "ec731ac5-49a1-4667-887a-b500f3732f2f"). InnerVolumeSpecName "kube-api-access-mfc6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:05:10.730226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.730181 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-xks-ca" pod "61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a" (UID: "61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:05:10.829730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.829693 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfc6p\" (UniqueName: \"kubernetes.io/projected/ec731ac5-49a1-4667-887a-b500f3732f2f-kube-api-access-mfc6p\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:10.829730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.829728 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8nmz\" (UniqueName: \"kubernetes.io/projected/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-kube-api-access-d8nmz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:10.829730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.829739 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec731ac5-49a1-4667-887a-b500f3732f2f-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:10.830033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:10.829763 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:05:11.381460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:11.381427 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:11.489545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:11.489515 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" Nov 01 16:05:11.490031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:11.489543 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb" event={"ID":"61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a","Type":"ContainerDied","Data":"e0c5cbfc8921b88076a021c7893eb73a4f6b26b7429b89f5d231c2dc460893b6"} Nov 01 16:05:11.490031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:11.489593 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c5cbfc8921b88076a021c7893eb73a4f6b26b7429b89f5d231c2dc460893b6" Nov 01 16:05:11.491141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:11.491116 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" event={"ID":"ec731ac5-49a1-4667-887a-b500f3732f2f","Type":"ContainerDied","Data":"b88eb50166a74393fe278d80acde56a48f25bf91c94c5fc196be24233a9ae77a"} Nov 01 16:05:11.491141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:11.491136 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh" Nov 01 16:05:11.491322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:11.491155 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b88eb50166a74393fe278d80acde56a48f25bf91c94c5fc196be24233a9ae77a" Nov 01 16:05:12.145646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:12.145606 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:12.947342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:12.947310 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:13.049602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:13.045784 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s"] Nov 01 16:05:13.062774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:13.059681 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd08a9d-s7z4s"] Nov 01 16:05:13.241981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:13.241893 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:05:13.686314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:13.686280 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:14.460170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:14.460137 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21b1b3f3-59a4-41b3-8225-7c6e2ed39fca" path="/var/lib/kubelet/pods/21b1b3f3-59a4-41b3-8225-7c6e2ed39fca/volumes" Nov 01 16:05:14.705449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:14.705416 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:15.508070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:15.508040 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:16.282743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:16.282704 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:16.932740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:16.932707 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:17.031527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:17.031493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:17.366075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:17.366034 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:17.735179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:17.735084 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:17.831728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:17.831694 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:18.309978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:18.309936 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:18.512796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:18.512721 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:18.614424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:18.614375 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:18.637220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:18.637185 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh"] Nov 01 16:05:18.647120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:18.647089 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-474cab5e-mwrbh"] Nov 01 16:05:18.700092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:18.700055 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:19.025735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:19.025633 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q"] Nov 01 16:05:19.031568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:19.031543 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61f137f8-t7b4q"] Nov 01 16:05:19.595001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:19.594968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:20.133519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:20.133473 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:20.460730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:20.460652 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c48ac04-6dda-407b-9803-4875988372ca" path="/var/lib/kubelet/pods/3c48ac04-6dda-407b-9803-4875988372ca/volumes" Nov 01 16:05:20.461211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:20.461195 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec731ac5-49a1-4667-887a-b500f3732f2f" path="/var/lib/kubelet/pods/ec731ac5-49a1-4667-887a-b500f3732f2f/volumes" Nov 01 16:05:20.516896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:20.516865 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:21.047956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.047919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:21.402786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.398029 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:21.402786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.401201 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7"] Nov 01 16:05:21.402786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.402312 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ec731ac5-49a1-4667-887a-b500f3732f2f" containerName="revoke-keys" Nov 01 16:05:21.402786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.402333 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec731ac5-49a1-4667-887a-b500f3732f2f" containerName="revoke-keys" Nov 01 16:05:21.402786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.402385 4573 cpu_manager.go:414] "containerMap: removing container" podUID="61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a" containerName="revoke-keys" Nov 01 16:05:21.402786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.402393 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a" containerName="revoke-keys" Nov 01 16:05:21.418587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.417525 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec731ac5-49a1-4667-887a-b500f3732f2f" containerName="revoke-keys" Nov 01 16:05:21.418587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.417592 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a" containerName="revoke-keys" Nov 01 16:05:21.434290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.434258 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7"] Nov 01 16:05:21.434466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.434373 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.438408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.438371 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-383fde-openshift-default-dockercfg-lxnqs\"" Nov 01 16:05:21.503452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.503342 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:21.511782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.505714 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.544026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.543986 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqt6n\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-kube-api-access-hqt6n\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.544742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.544717 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.544995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.544977 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.545205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.545187 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.549939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.549910 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.556776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.550190 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.557198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.557173 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.564771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.557556 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.681258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.677635 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.681533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.681512 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.681698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.681683 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.681832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.681817 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.682106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.682090 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.682339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.682325 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.682502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.682484 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqt6n\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-kube-api-access-hqt6n\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.682653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.682640 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.686033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.686007 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.686706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.686682 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.687122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.687103 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-credential-socket\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.687554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.687530 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-data\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.702125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.701392 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istiod-ca-cert\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.703217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.703132 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-certs\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.703692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.703665 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-podinfo\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.705883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.705199 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-envoy\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.723234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.723207 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-token\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.723816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.723788 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqt6n\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-kube-api-access-hqt6n\") pod \"e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.771162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.771130 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:21.996193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:21.996106 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:22.087606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.086876 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:22.090409 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:05:22.090377 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode28d462e_43f1_4063_94a5_b4bedac2f2b5.slice/crio-1a44191c880e90d802fe901736f569590501164006265c1fa18fe529cfe7167b WatchSource:0}: Error finding container 1a44191c880e90d802fe901736f569590501164006265c1fa18fe529cfe7167b: Status 404 returned error can't find the container with id 1a44191c880e90d802fe901736f569590501164006265c1fa18fe529cfe7167b Nov 01 16:05:22.091124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.091102 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7"] Nov 01 16:05:22.093268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.093239 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:22.093347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.093297 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:22.093347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.093330 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:05:22.279785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.260830 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:22.423623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.423594 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:22.425792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.425737 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb"] Nov 01 16:05:22.434993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.434955 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9fd01d20-dbgkb"] Nov 01 16:05:22.463534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.463499 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a" path="/var/lib/kubelet/pods/61bd5a5c-dc2c-42ed-92a2-c1a0eb90507a/volumes" Nov 01 16:05:22.519671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.519583 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:22.631249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.628638 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:22.867771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.867099 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" event={"ID":"e28d462e-43f1-4063-94a5-b4bedac2f2b5","Type":"ContainerStarted","Data":"91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a"} Nov 01 16:05:22.867771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.867243 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" event={"ID":"e28d462e-43f1-4063-94a5-b4bedac2f2b5","Type":"ContainerStarted","Data":"1a44191c880e90d802fe901736f569590501164006265c1fa18fe529cfe7167b"} Nov 01 16:05:22.925776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:22.925253 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podStartSLOduration=1.925231393 podStartE2EDuration="1.925231393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:22.890847035 +0000 UTC m=+1060.594012971" watchObservedRunningTime="2026-11-01 16:05:22.925231393 +0000 UTC m=+1060.628397331" Nov 01 16:05:23.373814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:23.373777 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:23.474473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:23.474443 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:23.771710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:23.771622 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:23.790192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:23.783936 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:23.912779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:23.910137 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:23.926827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:23.926624 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:05:24.070871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:24.067009 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f"] Nov 01 16:05:24.082573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:24.082521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:24.089698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:24.089614 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9472f900-zn84f"] Nov 01 16:05:24.325942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:24.325847 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:24.411559 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:05:24.407954 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:05:24.460475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:24.460443 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="287fbc1d-571f-41ae-afb9-251f2da0d568" path="/var/lib/kubelet/pods/287fbc1d-571f-41ae-afb9-251f2da0d568/volumes" Nov 01 16:05:24.596982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:24.596864 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:25.423010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:25.422968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:25.503167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:25.503124 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:26.198868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:26.198828 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:26.539543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:26.539448 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:26.641697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:26.641657 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:27.147256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:27.147220 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d"] Nov 01 16:05:27.153044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:27.153019 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a862ace3-5c94d"] Nov 01 16:05:27.301800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:27.301732 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:28.176942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:28.176910 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:28.462308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:28.462229 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab8c178-c15d-4009-a49d-611014f91f05" path="/var/lib/kubelet/pods/6ab8c178-c15d-4009-a49d-611014f91f05/volumes" Nov 01 16:05:29.043207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:29.043173 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:29.884710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:29.884677 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:30.021785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:30.021732 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:30.762133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:30.762101 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:30.863504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:30.863468 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:31.146843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:31.146810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:31.243488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:31.243456 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:31.868921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:31.868888 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:32.584588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:32.584544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:32.707776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:32.699620 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:32.820012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:32.819970 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:33.489095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:33.489059 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:34.172088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:34.172053 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:34.826038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:34.825999 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:35.640253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:35.640204 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:36.026385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:36.026276 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs"] Nov 01 16:05:36.032299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:36.032266 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h"] Nov 01 16:05:36.038288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:36.038265 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1897a7f-6vmzs"] Nov 01 16:05:36.044250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:36.044225 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-045ca2d2-trv4h"] Nov 01 16:05:36.345536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:36.345442 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:36.471598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:36.471555 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67df2dc9-d8e1-4566-a322-7895480f22ae" path="/var/lib/kubelet/pods/67df2dc9-d8e1-4566-a322-7895480f22ae/volumes" Nov 01 16:05:36.472330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:36.472304 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e39bf32-1293-4995-90a6-53b547f4aca2" path="/var/lib/kubelet/pods/9e39bf32-1293-4995-90a6-53b547f4aca2/volumes" Nov 01 16:05:37.032459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:37.032420 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n"] Nov 01 16:05:37.043215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:37.043183 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7"] Nov 01 16:05:37.052841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:37.052803 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa5f90da-p955n"] Nov 01 16:05:37.059809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:37.059779 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-981ffd63-w6jc7"] Nov 01 16:05:37.087770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:37.087721 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:37.187602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:37.187566 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:37.295827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:37.289527 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:38.091557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:38.091328 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:38.459848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:38.459814 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a544fe-3957-48ef-a092-1cc47533cf92" path="/var/lib/kubelet/pods/c4a544fe-3957-48ef-a092-1cc47533cf92/volumes" Nov 01 16:05:38.460351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:38.460336 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5bc0b55-74f0-4bf6-a338-24565fe9fa2c" path="/var/lib/kubelet/pods/f5bc0b55-74f0-4bf6-a338-24565fe9fa2c/volumes" Nov 01 16:05:38.973820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:38.973786 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:39.034612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:39.034579 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w"] Nov 01 16:05:39.040195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:39.040161 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f41548c3-pwv7w"] Nov 01 16:05:39.829888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:39.829851 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:40.459085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:40.459052 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0268b3d0-bccd-477a-96bf-91ddd8f89bc1" path="/var/lib/kubelet/pods/0268b3d0-bccd-477a-96bf-91ddd8f89bc1/volumes" Nov 01 16:05:40.550361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:40.550329 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:41.287343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:41.287313 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:42.310494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:42.310447 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:43.068305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:43.068264 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:43.764021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:43.763940 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:44.464505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:44.464471 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:45.163564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:45.163529 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:45.293668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:45.293613 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:45.721572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:45.721502 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:45.877200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:45.877158 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:46.267527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:46.267481 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:46.487679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:46.487646 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:46.840824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:46.840792 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:47.670019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:47.669983 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:48.464037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:48.464006 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:49.025185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:49.025152 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd"] Nov 01 16:05:49.030445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:49.030421 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbaaea25-2mcrd"] Nov 01 16:05:49.091063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:49.091028 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:49.895298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:49.895264 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:50.039266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:50.039233 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:50.058891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:50.058851 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t"] Nov 01 16:05:50.066282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:50.066252 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e113b817-7994t"] Nov 01 16:05:50.338743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:50.338660 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:50.459045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:50.459010 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23895580-3977-4307-a7d5-66cf8f425666" path="/var/lib/kubelet/pods/23895580-3977-4307-a7d5-66cf8f425666/volumes" Nov 01 16:05:50.459493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:50.459479 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52a838aa-0c34-4c19-8764-dd2054bc0b78" path="/var/lib/kubelet/pods/52a838aa-0c34-4c19-8764-dd2054bc0b78/volumes" Nov 01 16:05:50.710135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:50.710097 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:50.861281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:50.861238 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:51.013277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:51.013190 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:51.339308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:51.339225 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:51.858431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:51.858397 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:51.973446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:51.973382 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:52.029064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.029027 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2"] Nov 01 16:05:52.038076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.038047 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:52.047886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.047187 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2"] Nov 01 16:05:52.077676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.077636 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:52.128395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.128318 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2tsw\" (UniqueName: \"kubernetes.io/projected/e4e06231-d9cd-4796-a0a5-ef40859df1e5-kube-api-access-k2tsw\") pod \"payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2\" (UID: \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:52.128615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.128597 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e06231-d9cd-4796-a0a5-ef40859df1e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2\" (UID: \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:52.135092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.135070 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-a9a967\"" Nov 01 16:05:52.236819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.230880 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2tsw\" (UniqueName: \"kubernetes.io/projected/e4e06231-d9cd-4796-a0a5-ef40859df1e5-kube-api-access-k2tsw\") pod \"payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2\" (UID: \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:52.236819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.231311 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e06231-d9cd-4796-a0a5-ef40859df1e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2\" (UID: \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:52.236819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.232276 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e06231-d9cd-4796-a0a5-ef40859df1e5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2\" (UID: \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:52.266266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.266233 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2tsw\" (UniqueName: \"kubernetes.io/projected/e4e06231-d9cd-4796-a0a5-ef40859df1e5-kube-api-access-k2tsw\") pod \"payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2\" (UID: \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:52.369838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.368037 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:52.739320 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:05:52.739277 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e06231_d9cd_4796_a0a5_ef40859df1e5.slice/crio-2c95887b1ed1dfc67be81c328fe3cfc6ad02707eb9fef168220364f7b7bb2d46 WatchSource:0}: Error finding container 2c95887b1ed1dfc67be81c328fe3cfc6ad02707eb9fef168220364f7b7bb2d46: Status 404 returned error can't find the container with id 2c95887b1ed1dfc67be81c328fe3cfc6ad02707eb9fef168220364f7b7bb2d46 Nov 01 16:05:52.741093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.741070 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2"] Nov 01 16:05:52.863164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:52.863133 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:53.497294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:53.497259 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" event={"ID":"e4e06231-d9cd-4796-a0a5-ef40859df1e5","Type":"ContainerStarted","Data":"a5be05cfad5fdd5555e520ab3301dcb1b288a5327a1e0ed562b937041c127369"} Nov 01 16:05:53.497842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:53.497316 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" event={"ID":"e4e06231-d9cd-4796-a0a5-ef40859df1e5","Type":"ContainerStarted","Data":"2c95887b1ed1dfc67be81c328fe3cfc6ad02707eb9fef168220364f7b7bb2d46"} Nov 01 16:05:53.497842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:53.497350 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:05:53.522020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:53.521957 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" podStartSLOduration=2.52194329 podStartE2EDuration="2.52194329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:05:53.513370207 +0000 UTC m=+1091.216536172" watchObservedRunningTime="2026-11-01 16:05:53.52194329 +0000 UTC m=+1091.225109227" Nov 01 16:05:53.662141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:53.662096 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:54.365563 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:05:54.365528 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:05:54.385244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:54.385212 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:54.914839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:54.914800 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:55.017986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:55.017944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:55.553104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:55.553066 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:56.165258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:56.165224 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:56.330523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:56.330490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:56.453824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:56.443093 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:57.065182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:57.065146 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:57.707872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:57.707840 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:58.342980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:58.342948 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:59.111226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:59.111190 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:59.211796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:59.211722 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:05:59.310352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:05:59.310313 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:00.099390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:00.099359 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:00.765555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:00.765523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:01.470616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:01.470581 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:02.162274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:02.162239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:02.837521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:02.837487 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:03.565478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.565448 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244"] Nov 01 16:06:03.567178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.567147 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:03.578407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.578383 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.588128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.588097 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244"] Nov 01 16:06:03.614346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.614316 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:06:03.614607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.614582 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:06:03.641596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.641565 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:03.698652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.698619 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.698830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.698668 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.698830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.698806 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwlq\" (UniqueName: \"kubernetes.io/projected/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-kube-api-access-xtwlq\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.754537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.754489 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:03.801711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.801675 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwlq\" (UniqueName: \"kubernetes.io/projected/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-kube-api-access-xtwlq\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.802061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.802038 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.802166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.802148 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.803391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.803211 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.809582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.809545 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.811148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.811123 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwlq\" (UniqueName: \"kubernetes.io/projected/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-kube-api-access-xtwlq\") pod \"maas-api-revoke-keys-e2e-mt-47e1515a-c5244\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:03.904219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:03.904184 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:04.243809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:04.241157 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244"] Nov 01 16:06:04.517953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:04.517911 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:06:04.681163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:04.681127 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" event={"ID":"333046ee-7f68-4f2b-bf5e-36a1b6cc558c","Type":"ContainerStarted","Data":"5f30dd3be833ba96d4de4ac5ad6d0669e42c90f1c446ae3b9f3400d4544dfa8d"} Nov 01 16:06:04.729022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:04.728991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:05.380333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:05.380297 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:05.691463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:05.691427 4573 generic.go:356] "Generic (PLEG): container finished" podID="333046ee-7f68-4f2b-bf5e-36a1b6cc558c" containerID="a481b9411844996d7e4db5bce5d12d35fef7ab7c97d9265be9095779610679df" exitCode=0 Nov 01 16:06:05.691951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:05.691522 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" event={"ID":"333046ee-7f68-4f2b-bf5e-36a1b6cc558c","Type":"ContainerDied","Data":"a481b9411844996d7e4db5bce5d12d35fef7ab7c97d9265be9095779610679df"} Nov 01 16:06:06.149841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.149808 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:06.895982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.895954 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:06.994195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.994105 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-xks-ca\") pod \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " Nov 01 16:06:06.994353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.994265 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-openshift-ca\") pod \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " Nov 01 16:06:06.994538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.994511 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-kube-api-access-xtwlq\" (UniqueName: \"kubernetes.io/projected/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-kube-api-access-xtwlq\") pod \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\" (UID: \"333046ee-7f68-4f2b-bf5e-36a1b6cc558c\") " Nov 01 16:06:06.994874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.994650 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-openshift-ca" pod "333046ee-7f68-4f2b-bf5e-36a1b6cc558c" (UID: "333046ee-7f68-4f2b-bf5e-36a1b6cc558c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:06:06.995538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.995514 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:06.996849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.996823 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-xks-ca" pod "333046ee-7f68-4f2b-bf5e-36a1b6cc558c" (UID: "333046ee-7f68-4f2b-bf5e-36a1b6cc558c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:06:06.996914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:06.996846 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-kube-api-access-xtwlq" pod "333046ee-7f68-4f2b-bf5e-36a1b6cc558c" (UID: "333046ee-7f68-4f2b-bf5e-36a1b6cc558c"). InnerVolumeSpecName "kube-api-access-xtwlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:07.068801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:07.068732 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:07.097404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:07.097364 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:07.097404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:07.097399 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtwlq\" (UniqueName: \"kubernetes.io/projected/333046ee-7f68-4f2b-bf5e-36a1b6cc558c-kube-api-access-xtwlq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:07.711993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:07.711964 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" Nov 01 16:06:07.712179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:07.711994 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244" event={"ID":"333046ee-7f68-4f2b-bf5e-36a1b6cc558c","Type":"ContainerDied","Data":"5f30dd3be833ba96d4de4ac5ad6d0669e42c90f1c446ae3b9f3400d4544dfa8d"} Nov 01 16:06:07.712179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:07.712050 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f30dd3be833ba96d4de4ac5ad6d0669e42c90f1c446ae3b9f3400d4544dfa8d" Nov 01 16:06:07.750933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:07.750904 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:08.546627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:08.546593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:09.300659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:09.300623 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:10.037774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:10.037515 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:10.667364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:10.667329 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:10.921245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:10.921157 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:11.563015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:11.562974 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:12.280835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:12.280799 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:12.981860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:12.981821 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:13.147847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.144623 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:13.537035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.536999 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m"] Nov 01 16:06:13.538188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.538157 4573 cpu_manager.go:414] "containerMap: removing container" podUID="333046ee-7f68-4f2b-bf5e-36a1b6cc558c" containerName="revoke-keys" Nov 01 16:06:13.538188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.538187 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="333046ee-7f68-4f2b-bf5e-36a1b6cc558c" containerName="revoke-keys" Nov 01 16:06:13.538426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.538412 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="333046ee-7f68-4f2b-bf5e-36a1b6cc558c" containerName="revoke-keys" Nov 01 16:06:13.549770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.549728 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m"] Nov 01 16:06:13.549936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.549876 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.554208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.554182 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-76063d10-openshift-default-dockercfg-8n7f7\"" Nov 01 16:06:13.570742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.570708 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:13.662904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.662865 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:13.689595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.689566 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz"] Nov 01 16:06:13.699157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.699123 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf"] Nov 01 16:06:13.700168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700136 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cb6111-43a7-45b9-a7c6-d9398af6e215-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.700319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700289 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.700382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700333 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.700485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700453 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.700668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700650 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.700719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700696 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.700780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700734 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.700831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700781 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.700910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.700833 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkzj8\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-kube-api-access-zkzj8\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.769140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.769106 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:13.806585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.806473 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.806936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.806913 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.807051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.807003 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.807115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.807046 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.807115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.807072 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.807211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.807123 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkzj8\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-kube-api-access-zkzj8\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.807211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.807198 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cb6111-43a7-45b9-a7c6-d9398af6e215-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.807325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.807307 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.807922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.807898 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.808118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.808103 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.808301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.808276 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.808599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.808573 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cb6111-43a7-45b9-a7c6-d9398af6e215-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.808684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.808136 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.809058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.809030 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.811328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.811303 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.811538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.811523 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.816941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.816919 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkzj8\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-kube-api-access-zkzj8\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.817936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.817917 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:13.863260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.863206 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:13.882439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.882411 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:14.004778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:13.991170 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:14.445979 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:06:14.445939 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cb6111_43a7_45b9_a7c6_d9398af6e215.slice/crio-c5d25d2ea09f00ee199718bd2bfc9d0ee655fee2834fb09153f4221b6bad62f0 WatchSource:0}: Error finding container c5d25d2ea09f00ee199718bd2bfc9d0ee655fee2834fb09153f4221b6bad62f0: Status 404 returned error can't find the container with id c5d25d2ea09f00ee199718bd2bfc9d0ee655fee2834fb09153f4221b6bad62f0 Nov 01 16:06:14.449383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:14.449331 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:14.449601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:14.449571 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:14.449776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:14.449596 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m"] Nov 01 16:06:14.449776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:14.449631 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:14.977199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:14.977150 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:06:14.980702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:14.980665 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" event={"ID":"65cb6111-43a7-45b9-a7c6-d9398af6e215","Type":"ContainerStarted","Data":"6c83f536776ba4f42224e9b51fb97cdfe8ff7934db80e8b327d543fcb4590f0b"} Nov 01 16:06:14.980955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:14.980921 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" event={"ID":"65cb6111-43a7-45b9-a7c6-d9398af6e215","Type":"ContainerStarted","Data":"c5d25d2ea09f00ee199718bd2bfc9d0ee655fee2834fb09153f4221b6bad62f0"} Nov 01 16:06:15.017443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:15.017392 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podStartSLOduration=2.017375649 podStartE2EDuration="2.017375649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:15.009417909 +0000 UTC m=+1112.712583847" watchObservedRunningTime="2026-11-01 16:06:15.017375649 +0000 UTC m=+1112.720541585" Nov 01 16:06:15.084864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:15.084828 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:15.883475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:15.883442 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:15.889282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:15.889257 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:15.992353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:15.992320 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:15.994218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:15.994197 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:15.995019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:15.994994 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:06:17.049622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:17.049593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:17.299827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:17.299719 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:17.397116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:17.397079 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:17.881773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:17.876394 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:18.516225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:18.516185 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:18.616764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:18.616718 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:18.702708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:18.700826 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" podUID="920c2275-5fde-431c-8c58-1ad35a704ef0" containerName="payload-processing" containerID="cri-o://bfb18c5738901d36f31464373c5573f351e5b9691239c9b95dab599e7937a972" gracePeriod=25 Nov 01 16:06:18.702708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:18.701692 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" podUID="3cd71d44-e9b5-4533-a65c-3956b09b75f0" containerName="payload-pre-processing" containerID="cri-o://381cf51c190802739c9be020a61259a9bb01eb42635b43fab360a62d383dfaa0" gracePeriod=25 Nov 01 16:06:19.052618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.052566 4573 generic.go:356] "Generic (PLEG): container finished" podID="3cd71d44-e9b5-4533-a65c-3956b09b75f0" containerID="381cf51c190802739c9be020a61259a9bb01eb42635b43fab360a62d383dfaa0" exitCode=0 Nov 01 16:06:19.052903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.052834 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" event={"ID":"3cd71d44-e9b5-4533-a65c-3956b09b75f0","Type":"ContainerDied","Data":"381cf51c190802739c9be020a61259a9bb01eb42635b43fab360a62d383dfaa0"} Nov 01 16:06:19.056955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.056925 4573 generic.go:356] "Generic (PLEG): container finished" podID="920c2275-5fde-431c-8c58-1ad35a704ef0" containerID="bfb18c5738901d36f31464373c5573f351e5b9691239c9b95dab599e7937a972" exitCode=0 Nov 01 16:06:19.057111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.057093 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" event={"ID":"920c2275-5fde-431c-8c58-1ad35a704ef0","Type":"ContainerDied","Data":"bfb18c5738901d36f31464373c5573f351e5b9691239c9b95dab599e7937a972"} Nov 01 16:06:19.309866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.309836 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:06:19.400055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.400019 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cd71d44-e9b5-4533-a65c-3956b09b75f0-kube-api-access-mk4lf\" (UniqueName: \"kubernetes.io/projected/3cd71d44-e9b5-4533-a65c-3956b09b75f0-kube-api-access-mk4lf\") pod \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\" (UID: \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\") " Nov 01 16:06:19.400319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.400296 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cd71d44-e9b5-4533-a65c-3956b09b75f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd71d44-e9b5-4533-a65c-3956b09b75f0-plugins-config-volume\") pod \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\" (UID: \"3cd71d44-e9b5-4533-a65c-3956b09b75f0\") " Nov 01 16:06:19.400709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.400678 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd71d44-e9b5-4533-a65c-3956b09b75f0-plugins-config-volume" pod "3cd71d44-e9b5-4533-a65c-3956b09b75f0" (UID: "3cd71d44-e9b5-4533-a65c-3956b09b75f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:06:19.401563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.401542 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd71d44-e9b5-4533-a65c-3956b09b75f0-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:19.402965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.402937 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd71d44-e9b5-4533-a65c-3956b09b75f0-kube-api-access-mk4lf" pod "3cd71d44-e9b5-4533-a65c-3956b09b75f0" (UID: "3cd71d44-e9b5-4533-a65c-3956b09b75f0"). InnerVolumeSpecName "kube-api-access-mk4lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:19.424539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.424514 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:06:19.458232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.458197 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:19.502458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.502419 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/920c2275-5fde-431c-8c58-1ad35a704ef0-kube-api-access-9sx87\" (UniqueName: \"kubernetes.io/projected/920c2275-5fde-431c-8c58-1ad35a704ef0-kube-api-access-9sx87\") pod \"920c2275-5fde-431c-8c58-1ad35a704ef0\" (UID: \"920c2275-5fde-431c-8c58-1ad35a704ef0\") " Nov 01 16:06:19.502627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.502472 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/920c2275-5fde-431c-8c58-1ad35a704ef0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920c2275-5fde-431c-8c58-1ad35a704ef0-plugins-config-volume\") pod \"920c2275-5fde-431c-8c58-1ad35a704ef0\" (UID: \"920c2275-5fde-431c-8c58-1ad35a704ef0\") " Nov 01 16:06:19.503249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.503205 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/920c2275-5fde-431c-8c58-1ad35a704ef0-plugins-config-volume" pod "920c2275-5fde-431c-8c58-1ad35a704ef0" (UID: "920c2275-5fde-431c-8c58-1ad35a704ef0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:06:19.503499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.503479 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk4lf\" (UniqueName: \"kubernetes.io/projected/3cd71d44-e9b5-4533-a65c-3956b09b75f0-kube-api-access-mk4lf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:19.503620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.503503 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/920c2275-5fde-431c-8c58-1ad35a704ef0-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:19.609038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.609001 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/920c2275-5fde-431c-8c58-1ad35a704ef0-kube-api-access-9sx87" pod "920c2275-5fde-431c-8c58-1ad35a704ef0" (UID: "920c2275-5fde-431c-8c58-1ad35a704ef0"). InnerVolumeSpecName "kube-api-access-9sx87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:19.708373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:19.708278 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sx87\" (UniqueName: \"kubernetes.io/projected/920c2275-5fde-431c-8c58-1ad35a704ef0-kube-api-access-9sx87\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:20.042701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.042594 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:20.092401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.092367 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" Nov 01 16:06:20.092401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.092388 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz" event={"ID":"920c2275-5fde-431c-8c58-1ad35a704ef0","Type":"ContainerDied","Data":"1925c2c4d14e64229c477b78cab8b0da90545cb5eeef729ac9618e07e601c511"} Nov 01 16:06:20.092638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.092472 4573 scope.go:122] "RemoveContainer" containerID="bfb18c5738901d36f31464373c5573f351e5b9691239c9b95dab599e7937a972" Nov 01 16:06:20.096863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.096840 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" Nov 01 16:06:20.097006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.096839 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf" event={"ID":"3cd71d44-e9b5-4533-a65c-3956b09b75f0","Type":"ContainerDied","Data":"aaa7a40ddcae5f5e18ec6713641a10b3c96d4d25d18074d8e306e7a7c5b70eaf"} Nov 01 16:06:20.147954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.147930 4573 scope.go:122] "RemoveContainer" containerID="381cf51c190802739c9be020a61259a9bb01eb42635b43fab360a62d383dfaa0" Nov 01 16:06:20.155222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.154231 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz"] Nov 01 16:06:20.180859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.180811 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-47e1515a-5df486c565-cqhgz"] Nov 01 16:06:20.198349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.198318 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf"] Nov 01 16:06:20.205694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.205661 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-47e1515a-5cf9b7c76c-2wlwf"] Nov 01 16:06:20.256982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.256951 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:20.459793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.459745 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cd71d44-e9b5-4533-a65c-3956b09b75f0" path="/var/lib/kubelet/pods/3cd71d44-e9b5-4533-a65c-3956b09b75f0/volumes" Nov 01 16:06:20.460354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:20.460331 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="920c2275-5fde-431c-8c58-1ad35a704ef0" path="/var/lib/kubelet/pods/920c2275-5fde-431c-8c58-1ad35a704ef0/volumes" Nov 01 16:06:21.078775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:21.075136 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:21.621889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:21.621845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:21.783189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:21.783152 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:21.818073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:21.818040 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244"] Nov 01 16:06:21.823866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:21.823837 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-47e1515a-c5244"] Nov 01 16:06:21.878286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:21.878196 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:21.954456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:21.954425 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:22.301124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:22.301035 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:22.486508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:22.486465 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="333046ee-7f68-4f2b-bf5e-36a1b6cc558c" path="/var/lib/kubelet/pods/333046ee-7f68-4f2b-bf5e-36a1b6cc558c/volumes" Nov 01 16:06:22.982500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:22.982465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:23.822800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:23.822692 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:24.366823 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:06:24.366744 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:06:24.479566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:24.479530 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:24.952936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:24.952895 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:06:24.953423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:24.952976 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:06:25.173106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:25.173061 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:25.858683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:25.858647 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:26.565720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:26.565687 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:26.666317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:26.666283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:27.334008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:27.333967 4573 generic.go:356] "Generic (PLEG): container finished" podID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerID="91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a" exitCode=0 Nov 01 16:06:27.334240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:27.334044 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" event={"ID":"e28d462e-43f1-4063-94a5-b4bedac2f2b5","Type":"ContainerDied","Data":"91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a"} Nov 01 16:06:27.334547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:27.334532 4573 scope.go:122] "RemoveContainer" containerID="91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a" Nov 01 16:06:27.336391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:27.336361 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:27.336489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:27.336436 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:27.336489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:27.336466 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:27.469030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:27.469005 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:28.264903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:28.261455 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:28.431788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:28.428674 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" event={"ID":"e28d462e-43f1-4063-94a5-b4bedac2f2b5","Type":"ContainerStarted","Data":"ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198"} Nov 01 16:06:28.768611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:28.768572 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:28.775774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:28.774246 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:06:28.776118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:28.776090 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 01 16:06:28.776379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:28.776298 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 01 16:06:28.902740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:28.902708 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:29.099445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:29.099362 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:29.197958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:29.197923 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:29.597562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:29.597512 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:29.691044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:29.691010 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:29.785776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:29.777888 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 01 16:06:29.785776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:29.777953 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 01 16:06:29.888057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:29.888025 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:30.027733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:30.027694 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:30.774221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:30.774183 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 01 16:06:30.774912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:30.774879 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 01 16:06:30.811842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:30.811809 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:30.911561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:30.911528 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:31.297304 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.297270 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd"] Nov 01 16:06:31.298047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.298030 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3cd71d44-e9b5-4533-a65c-3956b09b75f0" containerName="payload-pre-processing" Nov 01 16:06:31.298119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.298049 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cd71d44-e9b5-4533-a65c-3956b09b75f0" containerName="payload-pre-processing" Nov 01 16:06:31.298119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.298066 4573 cpu_manager.go:414] "containerMap: removing container" podUID="920c2275-5fde-431c-8c58-1ad35a704ef0" containerName="payload-processing" Nov 01 16:06:31.298119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.298071 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="920c2275-5fde-431c-8c58-1ad35a704ef0" containerName="payload-processing" Nov 01 16:06:31.298244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.298233 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cd71d44-e9b5-4533-a65c-3956b09b75f0" containerName="payload-pre-processing" Nov 01 16:06:31.298278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.298250 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="920c2275-5fde-431c-8c58-1ad35a704ef0" containerName="payload-processing" Nov 01 16:06:31.302477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.302449 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.310832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.310801 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-a9a967-openshift-default-dockercfg-8bwnt\"" Nov 01 16:06:31.314918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.314892 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd"] Nov 01 16:06:31.445421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.445383 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.445619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.445439 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.445619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.445460 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.445717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.445691 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.445801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.445737 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.445907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.445887 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.446031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.446014 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b6d892-a51a-4141-aaf2-a5a0940e43db-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.446088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.446041 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv47m\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-kube-api-access-fv47m\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.446133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.446121 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.547954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.547852 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.547954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.547897 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.547954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.547920 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.547983 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b6d892-a51a-4141-aaf2-a5a0940e43db-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.547999 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv47m\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-kube-api-access-fv47m\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.548044 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.548142 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.548189 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.548217 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.548344 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.548360 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.548673 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b6d892-a51a-4141-aaf2-a5a0940e43db-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.548900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.548883 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.549052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.549032 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.550722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.550698 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.550840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.550822 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.555648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.555616 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv47m\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-kube-api-access-fv47m\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.555887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.555869 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.582648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.582608 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:31.622659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.622627 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:31.796516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.796479 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 01 16:06:31.797228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:31.797191 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 01 16:06:32.044788 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:06:32.044725 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b6d892_a51a_4141_aaf2_a5a0940e43db.slice/crio-69b49e976241549fae73952b446d379290a19804a323f81606b87e441b105a0c WatchSource:0}: Error finding container 69b49e976241549fae73952b446d379290a19804a323f81606b87e441b105a0c: Status 404 returned error can't find the container with id 69b49e976241549fae73952b446d379290a19804a323f81606b87e441b105a0c Nov 01 16:06:32.045213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.045186 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd"] Nov 01 16:06:32.047035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.046981 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.047144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.047058 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.047144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.047104 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:32.271774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.270641 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:32.559326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.559285 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" event={"ID":"42b6d892-a51a-4141-aaf2-a5a0940e43db","Type":"ContainerStarted","Data":"5440789990a73aed36ac74c863c400e7d8b859595bc1a25ae973aebc66c8caec"} Nov 01 16:06:32.559534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.559342 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" event={"ID":"42b6d892-a51a-4141-aaf2-a5a0940e43db","Type":"ContainerStarted","Data":"69b49e976241549fae73952b446d379290a19804a323f81606b87e441b105a0c"} Nov 01 16:06:32.608444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.608390 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" podStartSLOduration=1.608374215 podStartE2EDuration="1.608374215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:32.60059082 +0000 UTC m=+1130.303756747" watchObservedRunningTime="2026-11-01 16:06:32.608374215 +0000 UTC m=+1130.311540150" Nov 01 16:06:32.623714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.623686 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:32.779204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.779167 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 01 16:06:32.779526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:32.779497 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 01 16:06:33.119829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:33.119732 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:33.627201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:33.627166 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 16:06:33.627406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:33.627230 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 16:06:33.784326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:33.784292 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body= Nov 01 16:06:33.784599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:33.784575 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" Nov 01 16:06:33.860609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:33.860563 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:34.623208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:34.623172 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 16:06:34.623718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:34.623228 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 16:06:34.761619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:34.761589 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:34.791327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:34.791290 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:06:34.791849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:34.791829 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:06:34.792588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:34.792566 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:06:35.094860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.089467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:35.461455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.461419 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n"] Nov 01 16:06:35.476311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.476279 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.488442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.488414 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-f47890-openshift-default-dockercfg-qpfmq\"" Nov 01 16:06:35.506767 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.506727 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n"] Nov 01 16:06:35.518693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.518658 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.562917 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.562969 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787148b3-4c00-43f9-b568-2cd1f994c3d9-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.563026 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.563057 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jwh5\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-kube-api-access-4jwh5\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.563176 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.563284 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.563327 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.563365 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.585774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.563413 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.620713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.620679 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:35.623828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.623802 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 16:06:35.624233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.623872 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 16:06:35.699777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.674109 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.728844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.705959 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787148b3-4c00-43f9-b568-2cd1f994c3d9-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.728844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.725303 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.730499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.730472 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787148b3-4c00-43f9-b568-2cd1f994c3d9-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.730851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.730830 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.731063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.731048 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jwh5\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-kube-api-access-4jwh5\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.731326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.731313 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.731576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.731563 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.731720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.731704 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.731885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.731871 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.732041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.732024 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.733156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.733133 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.733855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.733835 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.737770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.734269 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.737952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.734428 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.751782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.749485 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jwh5\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-kube-api-access-4jwh5\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.777776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.774496 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.777776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.774627 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:35.785600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.779242 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:35.824787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:35.812013 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:36.197864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.197786 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:36.300548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.300513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:36.355277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.355238 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n"] Nov 01 16:06:36.356240 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:06:36.356203 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787148b3_4c00_43f9_b568_2cd1f994c3d9.slice/crio-8c0cc56e25ab9acee1bcd2b827cc866015173beeeafdd73b8dc175f7fdeaf852 WatchSource:0}: Error finding container 8c0cc56e25ab9acee1bcd2b827cc866015173beeeafdd73b8dc175f7fdeaf852: Status 404 returned error can't find the container with id 8c0cc56e25ab9acee1bcd2b827cc866015173beeeafdd73b8dc175f7fdeaf852 Nov 01 16:06:36.359117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.359078 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:36.359215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.359167 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:36.359267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.359212 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:36.594772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.594351 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:36.623452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.623421 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 16:06:36.623761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.623691 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 16:06:36.840740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.840657 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:36.844885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.844835 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" event={"ID":"787148b3-4c00-43f9-b568-2cd1f994c3d9","Type":"ContainerStarted","Data":"2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89"} Nov 01 16:06:36.845052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.844896 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" event={"ID":"787148b3-4c00-43f9-b568-2cd1f994c3d9","Type":"ContainerStarted","Data":"8c0cc56e25ab9acee1bcd2b827cc866015173beeeafdd73b8dc175f7fdeaf852"} Nov 01 16:06:36.906183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.906113 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podStartSLOduration=1.906092058 podStartE2EDuration="1.906092058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:36.899244636 +0000 UTC m=+1134.602410574" watchObservedRunningTime="2026-11-01 16:06:36.906092058 +0000 UTC m=+1134.609257996" Nov 01 16:06:36.929285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:36.929250 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:37.037789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:37.036916 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:37.623022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:37.622987 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 16:06:37.623236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:37.623050 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 16:06:37.813157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:37.813121 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:37.818887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:37.818861 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:37.853345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:37.853317 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:37.854233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:37.854212 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:06:38.092912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:38.092782 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:38.196575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:38.196546 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:38.624050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:38.624017 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 01 16:06:38.624264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:38.624068 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 01 16:06:39.271848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.271796 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:39.388778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.383295 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:39.628516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.628482 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:39.629420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.629028 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:39.630206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.630181 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:06:39.665506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.665471 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:39.845584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.845550 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn"] Nov 01 16:06:39.853047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.853018 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:39.854032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.854009 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn"] Nov 01 16:06:39.859581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.859557 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:06:39.859951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:39.859931 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:06:40.036607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.036514 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.036607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.036561 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp7xm\" (UniqueName: \"kubernetes.io/projected/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-kube-api-access-jp7xm\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.036851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.036797 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.139014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.138975 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.139216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.139033 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp7xm\" (UniqueName: \"kubernetes.io/projected/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-kube-api-access-jp7xm\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.139216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.139196 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.139833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.139810 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.141619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.141589 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.146819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.146767 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp7xm\" (UniqueName: \"kubernetes.io/projected/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-kube-api-access-jp7xm\") pod \"maas-api-revoke-sub-0eaccb92-498jn\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.180048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.180012 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:40.399604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.399573 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:40.484487 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:06:40.484448 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c00c4c_4bdf_4f5f_9b40_a2c1ee8e4098.slice/crio-662155fb83e81b5ceedca4297ef39f3beb124a0478a5b5725f63940a0a1c2b55 WatchSource:0}: Error finding container 662155fb83e81b5ceedca4297ef39f3beb124a0478a5b5725f63940a0a1c2b55: Status 404 returned error can't find the container with id 662155fb83e81b5ceedca4297ef39f3beb124a0478a5b5725f63940a0a1c2b55 Nov 01 16:06:40.485474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.485438 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn"] Nov 01 16:06:40.641774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.640910 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:40.839283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.839237 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:40.954021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:40.953970 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" event={"ID":"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098","Type":"ContainerStarted","Data":"662155fb83e81b5ceedca4297ef39f3beb124a0478a5b5725f63940a0a1c2b55"} Nov 01 16:06:41.648042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:41.648005 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:41.966897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:41.966861 4573 generic.go:356] "Generic (PLEG): container finished" podID="f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098" containerID="2ce533b2958f03378e68fbfede326c0fa6f8860e755dfefa9356dfa28d30ce63" exitCode=0 Nov 01 16:06:41.967100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:41.966948 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" event={"ID":"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098","Type":"ContainerDied","Data":"2ce533b2958f03378e68fbfede326c0fa6f8860e755dfefa9356dfa28d30ce63"} Nov 01 16:06:42.353022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:42.352942 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:42.532841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:42.532809 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:43.157595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.157567 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:43.205005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.204969 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:43.266976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.266403 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-openshift-ca\") pod \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " Nov 01 16:06:43.266976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.266533 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-kube-api-access-jp7xm\" (UniqueName: \"kubernetes.io/projected/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-kube-api-access-jp7xm\") pod \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " Nov 01 16:06:43.266976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.266801 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-xks-ca\") pod \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\" (UID: \"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098\") " Nov 01 16:06:43.292487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.292443 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-openshift-ca" pod "f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098" (UID: "f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:06:43.308506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.308455 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-xks-ca" pod "f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098" (UID: "f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:06:43.308817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.308770 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-kube-api-access-jp7xm" pod "f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098" (UID: "f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098"). InnerVolumeSpecName "kube-api-access-jp7xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:06:43.369186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.369153 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:43.369186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.369184 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:43.369398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.369196 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp7xm\" (UniqueName: \"kubernetes.io/projected/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098-kube-api-access-jp7xm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:06:43.474259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.474226 4573 scope.go:122] "RemoveContainer" containerID="63c8e71cc0e3cfeb3b384f253b85487f9f97074b47dd5f5d1dd3c84832dfdd89" Nov 01 16:06:43.486883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.486852 4573 scope.go:122] "RemoveContainer" containerID="09331a4a88221e34c8fb7dd6c24b9cc656a0f70d56f57f6aa74af4700b41529d" Nov 01 16:06:43.502658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.502628 4573 scope.go:122] "RemoveContainer" containerID="80d0c88fa969af725a83e9bf810dbd7a2367b4b8bcf9bf9ed0d1225a054d9a21" Nov 01 16:06:43.519512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.519488 4573 scope.go:122] "RemoveContainer" containerID="b395e4f898e39520352af0d72ac54efff19e89b0aa98bdc064516f5e2f7446f1" Nov 01 16:06:43.531623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.531600 4573 scope.go:122] "RemoveContainer" containerID="5745bacde7ccdadd2266b0e9224e662f6fc2aa1f1ed93b0589bfbed0803879d9" Nov 01 16:06:43.547131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.547101 4573 scope.go:122] "RemoveContainer" containerID="443af5c3b62fef7e13989b9ae0066d27db684af7d528966168894c28135098ef" Nov 01 16:06:43.558434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.558411 4573 scope.go:122] "RemoveContainer" containerID="2bf8385e90728f7f244f5d989b217718f0258f592c3c2d0b096da7687f202a4c" Nov 01 16:06:43.568556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.568536 4573 scope.go:122] "RemoveContainer" containerID="5a7818597638b23364abac4a396d24e1a5dbc12d6123c5be7c01d3401fe7c53e" Nov 01 16:06:43.581044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.581020 4573 scope.go:122] "RemoveContainer" containerID="d9d2cf8d9859f9a08d0cba9d692ccbd89979b00ef008da18cac1b7c4a6c518ec" Nov 01 16:06:43.591076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.591053 4573 scope.go:122] "RemoveContainer" containerID="13f8300f3806945534375fe4a3cc3b6647802a565a83c69e01976a34e48dc813" Nov 01 16:06:43.986918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.986891 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" Nov 01 16:06:43.987111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.986920 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn" event={"ID":"f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098","Type":"ContainerDied","Data":"662155fb83e81b5ceedca4297ef39f3beb124a0478a5b5725f63940a0a1c2b55"} Nov 01 16:06:43.987111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:43.986972 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="662155fb83e81b5ceedca4297ef39f3beb124a0478a5b5725f63940a0a1c2b55" Nov 01 16:06:44.000464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:44.000350 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:44.935278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:44.935241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:45.671490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:45.671462 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:46.454368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:46.454310 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:46.552189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:46.552125 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:47.172102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:47.172067 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:48.086359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:48.086325 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:48.857431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:48.857396 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:49.586208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:49.586174 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:49.685462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:49.685426 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:49.797778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:49.795841 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:50.034944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.034902 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:50.323844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.323701 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:50.547769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.547731 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:50.926022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.925985 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt"] Nov 01 16:06:50.927113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.927095 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098" containerName="revoke-keys" Nov 01 16:06:50.927182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.927116 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098" containerName="revoke-keys" Nov 01 16:06:50.927273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.927254 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098" containerName="revoke-keys" Nov 01 16:06:50.935951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.935927 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.950343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.950320 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt"] Nov 01 16:06:50.958130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.958110 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-61bb54-openshift-default-dockercfg-bxttg\"" Nov 01 16:06:50.988815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.988770 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:50.990335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.990309 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-podinfo\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.990527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.990492 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-token\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.990618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.990600 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-socket\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.990832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.990806 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-data\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.990945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.990872 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-certs\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.991071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.990951 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-envoy\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.991157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.991073 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-credential-socket\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.991157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.991130 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6nx\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-kube-api-access-4n6nx\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:50.991269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:50.991170 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30902ce9-0a12-45d5-a61a-bd13ed695adb-istiod-ca-cert\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.100410 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-token\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.100515 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-socket\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.100629 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-data\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.100683 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-certs\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.100712 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-envoy\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.100821 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-credential-socket\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.100879 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6nx\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-kube-api-access-4n6nx\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.100924 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30902ce9-0a12-45d5-a61a-bd13ed695adb-istiod-ca-cert\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.103027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.101038 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-podinfo\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.129676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.128467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:51.129676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.129248 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-certs\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.129676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.129348 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-credential-socket\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.137648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.130254 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-data\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.137648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.131010 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30902ce9-0a12-45d5-a61a-bd13ed695adb-istiod-ca-cert\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.158865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.146687 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-socket\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.203950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.201316 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-token\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.206662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.206418 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-envoy\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.207853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.207835 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-podinfo\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.208726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.208703 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6nx\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-kube-api-access-4n6nx\") pod \"e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.257731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.257695 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:51.582922 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:06:51.582892 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30902ce9_0a12_45d5_a61a_bd13ed695adb.slice/crio-7bae41f8af375db7653a52dac594e96a4f789879cc4402092fdf6a9bb290341e WatchSource:0}: Error finding container 7bae41f8af375db7653a52dac594e96a4f789879cc4402092fdf6a9bb290341e: Status 404 returned error can't find the container with id 7bae41f8af375db7653a52dac594e96a4f789879cc4402092fdf6a9bb290341e Nov 01 16:06:51.584446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.584416 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt"] Nov 01 16:06:51.585425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.585391 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:51.585511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.585454 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:51.585511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.585482 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:06:51.640872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:51.640838 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:52.199455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:52.199422 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" event={"ID":"30902ce9-0a12-45d5-a61a-bd13ed695adb","Type":"ContainerStarted","Data":"8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6"} Nov 01 16:06:52.200006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:52.199471 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" event={"ID":"30902ce9-0a12-45d5-a61a-bd13ed695adb","Type":"ContainerStarted","Data":"7bae41f8af375db7653a52dac594e96a4f789879cc4402092fdf6a9bb290341e"} Nov 01 16:06:52.223232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:52.223167 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podStartSLOduration=2.223148731 podStartE2EDuration="2.223148731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:52.217530402 +0000 UTC m=+1149.920696341" watchObservedRunningTime="2026-11-01 16:06:52.223148731 +0000 UTC m=+1149.926314665" Nov 01 16:06:52.258134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:52.258109 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:52.264359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:52.264338 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:52.366228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:52.366194 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:53.136707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:53.136668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:53.253947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:53.253672 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:53.263744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:53.263712 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:06:54.056525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:54.056493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:54.366191 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:06:54.366154 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:06:54.659538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:54.659456 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:54.768836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:54.768798 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:55.033435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.033302 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:55.842264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.842234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:55.892018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.891983 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv"] Nov 01 16:06:55.896998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.896981 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:55.901949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.901925 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-61bb54\"" Nov 01 16:06:55.902083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.901965 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-61bb54-dockercfg-stwmb\"" Nov 01 16:06:55.906403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.906380 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv"] Nov 01 16:06:55.930834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.930793 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:55.973426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.973391 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnnf\" (UniqueName: \"kubernetes.io/projected/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-kube-api-access-ddnnf\") pod \"payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv\" (UID: \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\") " pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:55.973628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:55.973437 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv\" (UID: \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\") " pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:56.098818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:56.082017 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnnf\" (UniqueName: \"kubernetes.io/projected/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-kube-api-access-ddnnf\") pod \"payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv\" (UID: \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\") " pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:56.098818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:56.082085 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv\" (UID: \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\") " pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:56.098818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:56.083290 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv\" (UID: \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\") " pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:56.160305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:56.111202 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnnf\" (UniqueName: \"kubernetes.io/projected/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-kube-api-access-ddnnf\") pod \"payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv\" (UID: \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\") " pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:56.284065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:56.278065 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:56.291897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:56.291388 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:56.691453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:56.691421 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv"] Nov 01 16:06:56.692584 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:06:56.692552 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda512c9e7_ddc2_4c13_93a3_6c828dbafce7.slice/crio-5c97d5cb4cb382ccdbec63acffa5253844dd969d1a852e2948d90b8d99138cf5 WatchSource:0}: Error finding container 5c97d5cb4cb382ccdbec63acffa5253844dd969d1a852e2948d90b8d99138cf5: Status 404 returned error can't find the container with id 5c97d5cb4cb382ccdbec63acffa5253844dd969d1a852e2948d90b8d99138cf5 Nov 01 16:06:57.039449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.039370 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:57.039847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.039465 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5"] Nov 01 16:06:57.045294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.045267 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bbf71bf-skbj5"] Nov 01 16:06:57.153554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.153515 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:57.346130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.346033 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" event={"ID":"a512c9e7-ddc2-4c13-93a3-6c828dbafce7","Type":"ContainerStarted","Data":"9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c"} Nov 01 16:06:57.346130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.346094 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" event={"ID":"a512c9e7-ddc2-4c13-93a3-6c828dbafce7","Type":"ContainerStarted","Data":"5c97d5cb4cb382ccdbec63acffa5253844dd969d1a852e2948d90b8d99138cf5"} Nov 01 16:06:57.346130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.346122 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:06:57.364353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.364305 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" podStartSLOduration=2.364291524 podStartE2EDuration="2.364291524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:57.361349503 +0000 UTC m=+1155.064515438" watchObservedRunningTime="2026-11-01 16:06:57.364291524 +0000 UTC m=+1155.067457459" Nov 01 16:06:57.840801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:57.840742 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:58.098256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.098181 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97"] Nov 01 16:06:58.103613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.103588 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.107745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.107714 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:06:58.108076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.108055 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:06:58.110214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.110193 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97"] Nov 01 16:06:58.203383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.203354 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.203560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.203479 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.203797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.203742 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ndd6\" (UniqueName: \"kubernetes.io/projected/82fe8fbe-81ae-4977-8fe5-72ee75780644-kube-api-access-2ndd6\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.304906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.304873 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ndd6\" (UniqueName: \"kubernetes.io/projected/82fe8fbe-81ae-4977-8fe5-72ee75780644-kube-api-access-2ndd6\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.305104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.304997 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.305104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.305072 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.305774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.305737 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.307402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.307381 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.314183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.314160 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ndd6\" (UniqueName: \"kubernetes.io/projected/82fe8fbe-81ae-4977-8fe5-72ee75780644-kube-api-access-2ndd6\") pod \"maas-api-revoke-sub-1b6afa92-ktt97\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.420808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.420738 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:06:58.462305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.462269 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6247475a-850f-47e2-8e6b-31f11e90e201" path="/var/lib/kubelet/pods/6247475a-850f-47e2-8e6b-31f11e90e201/volumes" Nov 01 16:06:58.674673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.674584 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:06:58.810194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:58.810165 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97"] Nov 01 16:06:59.366596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:59.366563 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" event={"ID":"82fe8fbe-81ae-4977-8fe5-72ee75780644","Type":"ContainerStarted","Data":"a114be7b9d0daccd0aadbb6a7679e01a8e8ce5d69d9acbeb01cb53cdbae366a8"} Nov 01 16:06:59.367073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:59.366615 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" event={"ID":"82fe8fbe-81ae-4977-8fe5-72ee75780644","Type":"ContainerStarted","Data":"754836f38af5ebe79c03830dd7afb214ebb0e6c2ee3af0e2491a07188cf33c2d"} Nov 01 16:06:59.383973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:59.383927 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" podStartSLOduration=1.383915013 podStartE2EDuration="1.383915013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:06:59.382720553 +0000 UTC m=+1157.085886501" watchObservedRunningTime="2026-11-01 16:06:59.383915013 +0000 UTC m=+1157.087080948" Nov 01 16:06:59.414454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:06:59.414423 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:00.150285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:00.150252 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:00.377354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:00.377322 4573 generic.go:356] "Generic (PLEG): container finished" podID="82fe8fbe-81ae-4977-8fe5-72ee75780644" containerID="a114be7b9d0daccd0aadbb6a7679e01a8e8ce5d69d9acbeb01cb53cdbae366a8" exitCode=0 Nov 01 16:07:00.377734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:00.377408 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" event={"ID":"82fe8fbe-81ae-4977-8fe5-72ee75780644","Type":"ContainerDied","Data":"a114be7b9d0daccd0aadbb6a7679e01a8e8ce5d69d9acbeb01cb53cdbae366a8"} Nov 01 16:07:00.714024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:00.713993 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:07:00.785076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:00.785047 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:01.574488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.574461 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:07:01.677483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.677382 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-xks-ca\") pod \"82fe8fbe-81ae-4977-8fe5-72ee75780644\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " Nov 01 16:07:01.677655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.677578 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fe8fbe-81ae-4977-8fe5-72ee75780644-kube-api-access-2ndd6\" (UniqueName: \"kubernetes.io/projected/82fe8fbe-81ae-4977-8fe5-72ee75780644-kube-api-access-2ndd6\") pod \"82fe8fbe-81ae-4977-8fe5-72ee75780644\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " Nov 01 16:07:01.677722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.677682 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-openshift-ca\") pod \"82fe8fbe-81ae-4977-8fe5-72ee75780644\" (UID: \"82fe8fbe-81ae-4977-8fe5-72ee75780644\") " Nov 01 16:07:01.678245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.678217 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-openshift-ca" pod "82fe8fbe-81ae-4977-8fe5-72ee75780644" (UID: "82fe8fbe-81ae-4977-8fe5-72ee75780644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:07:01.678638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.678612 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:07:01.680642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.680611 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-xks-ca" pod "82fe8fbe-81ae-4977-8fe5-72ee75780644" (UID: "82fe8fbe-81ae-4977-8fe5-72ee75780644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:07:01.680810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.680638 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fe8fbe-81ae-4977-8fe5-72ee75780644-kube-api-access-2ndd6" pod "82fe8fbe-81ae-4977-8fe5-72ee75780644" (UID: "82fe8fbe-81ae-4977-8fe5-72ee75780644"). InnerVolumeSpecName "kube-api-access-2ndd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:07:01.695481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.695447 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:01.784773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.781573 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ndd6\" (UniqueName: \"kubernetes.io/projected/82fe8fbe-81ae-4977-8fe5-72ee75780644-kube-api-access-2ndd6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:07:01.784773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:01.781610 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82fe8fbe-81ae-4977-8fe5-72ee75780644-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:07:02.550776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:02.539242 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" Nov 01 16:07:02.615778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:02.594042 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97" event={"ID":"82fe8fbe-81ae-4977-8fe5-72ee75780644","Type":"ContainerDied","Data":"754836f38af5ebe79c03830dd7afb214ebb0e6c2ee3af0e2491a07188cf33c2d"} Nov 01 16:07:02.616588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:02.616555 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="754836f38af5ebe79c03830dd7afb214ebb0e6c2ee3af0e2491a07188cf33c2d" Nov 01 16:07:02.834571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:02.834483 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:04.337774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:04.337431 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:04.425774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:04.424344 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:04.885775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:04.882125 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:05.084777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.074155 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf"] Nov 01 16:07:05.113776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.106915 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e50165da-2x4xf"] Nov 01 16:07:05.450139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.449222 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:05.516776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.515829 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:05.574774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.566668 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df"] Nov 01 16:07:05.602778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.592440 4573 cpu_manager.go:414] "containerMap: removing container" podUID="82fe8fbe-81ae-4977-8fe5-72ee75780644" containerName="revoke-keys" Nov 01 16:07:05.602778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.592478 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fe8fbe-81ae-4977-8fe5-72ee75780644" containerName="revoke-keys" Nov 01 16:07:05.602778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.592782 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fe8fbe-81ae-4977-8fe5-72ee75780644" containerName="revoke-keys" Nov 01 16:07:05.708011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.701008 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:05.734469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.713002 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-734859-openshift-default-dockercfg-gwnw2\"" Nov 01 16:07:05.815137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.789076 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:05.815137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.801045 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df"] Nov 01 16:07:05.839774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.827987 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:05.942723 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.000930 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.025966 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.011059 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.012107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.026142 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24s6\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-kube-api-access-z24s6\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.026217 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.026388 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.026477 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.026520 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.026547 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a241e7-c6e7-4a85-80d4-906fbde4253d-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.030820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.026669 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.051377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.050816 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.084192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.078947 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:07:06.130299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.130256 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.131147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.131120 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.133071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.133048 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.133950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.133929 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24s6\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-kube-api-access-z24s6\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.149138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.149108 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.149981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.149949 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.150529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.150510 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.164859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.164832 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.192777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.169180 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a241e7-c6e7-4a85-80d4-906fbde4253d-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.193285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.133785 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.194836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.150901 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-data\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.216276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.130974 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-credential-socket\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.229938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.150393 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-certs\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.230158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.202723 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-podinfo\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.230258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.206533 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a241e7-c6e7-4a85-80d4-906fbde4253d-istiod-ca-cert\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.230393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.213507 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-token\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.257250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.257212 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-envoy\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.258191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.258099 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24s6\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-kube-api-access-z24s6\") pod \"e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.331325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.331236 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:06.504796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.490902 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:06.650338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.650300 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af195cd-dfeb-4ba2-b4d2-f7f832c94938" path="/var/lib/kubelet/pods/1af195cd-dfeb-4ba2-b4d2-f7f832c94938/volumes" Nov 01 16:07:06.706782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.702008 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:06.916894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.916713 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:07:06.919745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:06.919724 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:07:07.049449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:07.021901 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:07:07.484506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:07.484456 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df"] Nov 01 16:07:07.500235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:07.500206 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:07.507546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:07.507507 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:07.508287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:07.508260 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:07.508463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:07.508442 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:08.357000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:08.356972 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:07:08.400649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:08.400612 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" event={"ID":"84a241e7-c6e7-4a85-80d4-906fbde4253d","Type":"ContainerStarted","Data":"f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe"} Nov 01 16:07:08.400825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:08.400674 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" event={"ID":"84a241e7-c6e7-4a85-80d4-906fbde4253d","Type":"ContainerStarted","Data":"789fa25be4c11a462e62e72aa2892626632335fae377aeca24deb028c689c5cc"} Nov 01 16:07:08.420450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:08.420399 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podStartSLOduration=3.420385458 podStartE2EDuration="3.420385458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:08.418264246 +0000 UTC m=+1166.121430184" watchObservedRunningTime="2026-11-01 16:07:08.420385458 +0000 UTC m=+1166.123551393" Nov 01 16:07:08.547776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:08.539868 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:08.803687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:08.803607 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:09.250044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:09.250009 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:09.336979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:09.336950 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:09.343050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:09.343025 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:09.409792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:09.409746 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:09.410642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:09.410624 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:07:09.693218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:09.693167 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:09.787026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:09.775660 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:10.192278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:10.192243 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:10.293471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:10.293438 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:10.433887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:10.427990 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:10.677802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:10.676903 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:10.776270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:10.776226 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:11.631801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:11.631759 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:11.927584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:11.927421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:12.042534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:12.042503 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8"] Nov 01 16:07:12.043010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:12.042983 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:12.050397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:12.050365 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d182e87-l8vs8"] Nov 01 16:07:12.467815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:12.467727 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc086bef-b8c6-4566-8e03-0d40c08ec484" path="/var/lib/kubelet/pods/cc086bef-b8c6-4566-8e03-0d40c08ec484/volumes" Nov 01 16:07:12.895777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:12.891636 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:13.615016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:13.614950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:13.733278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:13.733221 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:14.090322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:14.090223 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:14.651271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:14.651223 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:14.795673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:14.795623 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:14.893586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:14.893548 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:15.800374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:15.800342 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:15.873135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:15.873107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:16.040224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.040188 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": read tcp 10.133.0.2:57924->10.133.0.144:15021: read: connection reset by peer" start-of-body= Nov 01 16:07:16.041010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.040974 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": read tcp 10.133.0.2:57924->10.133.0.144:15021: read: connection reset by peer" Nov 01 16:07:16.332170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.332080 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db"] Nov 01 16:07:16.336904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.336883 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.345739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.345715 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db"] Nov 01 16:07:16.361226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361194 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.361417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361233 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.361417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361334 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6235185-3f02-4ff5-b304-1fa951c09248-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.361417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361364 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.361417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361407 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.361659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361428 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr9sw\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-kube-api-access-sr9sw\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.361659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361536 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.361659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361631 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6235185-3f02-4ff5-b304-1fa951c09248-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.361943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.361921 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.463363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463328 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6235185-3f02-4ff5-b304-1fa951c09248-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.463602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463577 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.463734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463718 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.463803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463769 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.463843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463827 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6235185-3f02-4ff5-b304-1fa951c09248-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.463887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463860 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.463926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463885 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.463926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463899 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.464029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463909 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr9sw\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-kube-api-access-sr9sw\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.464029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.463979 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.464195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.464163 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.464296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.464273 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.464385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.464315 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.464592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.464572 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6235185-3f02-4ff5-b304-1fa951c09248-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.466314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.466291 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.466512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.466496 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6235185-3f02-4ff5-b304-1fa951c09248-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.471459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.471440 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.471557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.471476 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr9sw\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-kube-api-access-sr9sw\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:16.511659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.511630 4573 generic.go:356] "Generic (PLEG): container finished" podID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerID="6c83f536776ba4f42224e9b51fb97cdfe8ff7934db80e8b327d543fcb4590f0b" exitCode=0 Nov 01 16:07:16.511929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.511707 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" event={"ID":"65cb6111-43a7-45b9-a7c6-d9398af6e215","Type":"ContainerDied","Data":"6c83f536776ba4f42224e9b51fb97cdfe8ff7934db80e8b327d543fcb4590f0b"} Nov 01 16:07:16.512286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.512271 4573 scope.go:122] "RemoveContainer" containerID="6c83f536776ba4f42224e9b51fb97cdfe8ff7934db80e8b327d543fcb4590f0b" Nov 01 16:07:16.514439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.514407 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:16.514538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.514516 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:16.514581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.514562 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:16.567633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.567598 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:16.664376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:16.664077 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:17.051362 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:07:17.051050 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6235185_3f02_4ff5_b304_1fa951c09248.slice/crio-6e9714d7b21651792680a9afe08be81cd5a2880886460e208ef24d9ee67f3b2e WatchSource:0}: Error finding container 6e9714d7b21651792680a9afe08be81cd5a2880886460e208ef24d9ee67f3b2e: Status 404 returned error can't find the container with id 6e9714d7b21651792680a9afe08be81cd5a2880886460e208ef24d9ee67f3b2e Nov 01 16:07:17.052562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.052533 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db"] Nov 01 16:07:17.055060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.055025 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:17.055157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.055108 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:17.055288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.055153 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:17.302666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.302575 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:17.533631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.532738 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" event={"ID":"e6235185-3f02-4ff5-b304-1fa951c09248","Type":"ContainerStarted","Data":"0ea0c27334103b1a780f41702e4bd2c6c71d6a8ec791f7cc89a97eff1efb652b"} Nov 01 16:07:17.533631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.532841 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" event={"ID":"e6235185-3f02-4ff5-b304-1fa951c09248","Type":"ContainerStarted","Data":"6e9714d7b21651792680a9afe08be81cd5a2880886460e208ef24d9ee67f3b2e"} Nov 01 16:07:17.537246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.536520 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" event={"ID":"65cb6111-43a7-45b9-a7c6-d9398af6e215","Type":"ContainerStarted","Data":"493ecf574aa1c08e7d0cc8d8c8fd873f8af785e11999a2670f2c9199c5897530"} Nov 01 16:07:17.555727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.555600 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" podStartSLOduration=1.555582236 podStartE2EDuration="1.555582236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:17.553141114 +0000 UTC m=+1175.256307051" watchObservedRunningTime="2026-11-01 16:07:17.555582236 +0000 UTC m=+1175.258748172" Nov 01 16:07:17.680773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.678840 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:17.884163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.884123 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:07:17.884613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.884479 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:07:17.884821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:17.884795 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 16:07:18.006462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:18.006429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:18.705777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:18.694470 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 16:07:18.705777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:18.694536 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 16:07:18.751698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:18.751662 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:18.979834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:18.964122 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:07:18.979834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:18.964184 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 16:07:19.582774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:19.582293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:19.705778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:19.705698 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 16:07:19.705986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:19.705803 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 16:07:19.898787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:19.898206 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:07:19.898787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:19.898277 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 16:07:20.298832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:20.288260 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:20.681776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:20.671189 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 16:07:20.681776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:20.671268 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 16:07:20.883215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:20.883162 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:07:20.883726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:20.883228 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 16:07:20.973856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:20.971049 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:21.687776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:21.687157 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 16:07:21.687776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:21.687218 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 16:07:21.710823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:21.709685 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:21.889776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:21.886466 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:07:21.889776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:21.886588 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 16:07:22.322081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:22.322040 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:22.716593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:22.716539 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 16:07:22.719916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:22.717880 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 16:07:22.897905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:22.897865 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:07:22.898476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:22.898439 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 16:07:23.011830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:23.011294 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:23.191802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:23.189182 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:23.500234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:23.500201 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:23.667253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:23.667215 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 01 16:07:23.667521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:23.667491 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 01 16:07:23.883774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:23.883644 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Nov 01 16:07:23.883774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:23.883705 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Nov 01 16:07:23.992145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:23.992111 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:24.369377 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:07:24.369342 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:07:24.800825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:24.797792 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:24.921453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:24.921418 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:24.941046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:24.940917 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:24.968012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:24.967982 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:07:24.990940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:24.990904 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:07:24.992556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:24.992529 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:07:24.993934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:24.993702 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:07:25.745780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:25.745736 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:25.915039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:25.915007 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:26.014282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:26.014204 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:26.857952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:26.857917 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:27.754454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:27.754419 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:28.420315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:28.420285 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:29.071559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:29.071519 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:29.875008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:29.874970 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:30.152503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:30.152418 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:30.558563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:30.558482 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:30.821903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:30.821820 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:30.921987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:30.921947 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:31.069879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:31.069845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:31.233332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:31.233291 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:31.332956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:31.332923 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:31.442443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:31.437247 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:32.143099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:32.143052 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:32.881654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:32.881616 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:33.046382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:33.046346 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx"] Nov 01 16:07:33.089973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:33.089941 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa6f3092-sgkcx"] Nov 01 16:07:33.898244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:33.898212 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:34.459447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:34.459414 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81312c68-0f0e-46be-8c7f-64147d6e23eb" path="/var/lib/kubelet/pods/81312c68-0f0e-46be-8c7f-64147d6e23eb/volumes" Nov 01 16:07:34.667880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:34.667845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:35.730648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:35.730612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:36.454569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:36.454532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:36.772709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:36.772626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:36.867571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:36.867537 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:37.512381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:37.512351 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:38.070087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:38.070052 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g"] Nov 01 16:07:38.076435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:38.076401 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2"] Nov 01 16:07:38.081825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:38.081799 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-26c559d0-b2m4g"] Nov 01 16:07:38.087296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:38.087272 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca58e7ae-tptf2"] Nov 01 16:07:38.279807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:38.278851 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:38.466643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:38.466615 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad054e7-5660-4df2-86ff-5fc8cd765683" path="/var/lib/kubelet/pods/7ad054e7-5660-4df2-86ff-5fc8cd765683/volumes" Nov 01 16:07:38.486819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:38.467517 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d74c586-6de5-4d16-8469-66ebaeb3b906" path="/var/lib/kubelet/pods/9d74c586-6de5-4d16-8469-66ebaeb3b906/volumes" Nov 01 16:07:39.032148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:39.032111 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm"] Nov 01 16:07:39.041636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:39.041604 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2ea38d6-tgsgm"] Nov 01 16:07:39.168651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:39.168617 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:39.879848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:39.879809 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:40.460715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:40.460681 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4d55bc-75a1-4f9e-a45c-69bf127fddc2" path="/var/lib/kubelet/pods/da4d55bc-75a1-4f9e-a45c-69bf127fddc2/volumes" Nov 01 16:07:40.719322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:40.719227 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:40.970661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:40.970576 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:41.458826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:41.458794 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:42.233773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:42.233257 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:42.332443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:42.332403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:42.444467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:42.444435 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:42.651666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:42.651631 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:07:42.657705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:42.657678 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:07:42.667239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:42.667216 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:07:42.670408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:42.670389 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:07:42.951139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:42.951054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:43.656227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:43.656194 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:43.770780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:43.764068 4573 scope.go:122] "RemoveContainer" containerID="f5cc9b4c424a0dc6cfee892ca5611f85fa369eabd2021fb0a2ca81e9b1084eb5" Nov 01 16:07:43.798171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:43.798140 4573 scope.go:122] "RemoveContainer" containerID="f217e16b6ba8f8c9e9d0a02dc8210167ee4fdd2c3de9208804fdb9a8565d5411" Nov 01 16:07:43.991972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:43.991885 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:44.027131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.027098 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js"] Nov 01 16:07:44.032856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.032832 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9"] Nov 01 16:07:44.038428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.038403 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eceb48c4-6d6js"] Nov 01 16:07:44.045029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.045004 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a8e125-4vjg9"] Nov 01 16:07:44.386108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.386071 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:44.487916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.487877 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba172767-ecd9-4ada-9030-87c5a01e6a99" path="/var/lib/kubelet/pods/ba172767-ecd9-4ada-9030-87c5a01e6a99/volumes" Nov 01 16:07:44.489009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.488983 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cadea4b4-8df3-4b9f-89f2-fd9769e28187" path="/var/lib/kubelet/pods/cadea4b4-8df3-4b9f-89f2-fd9769e28187/volumes" Nov 01 16:07:44.504378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.504338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:44.827186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.827098 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6"] Nov 01 16:07:44.852628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.850145 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6"] Nov 01 16:07:44.852628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.850273 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.854595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.854561 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-f01de5-openshift-default-dockercfg-czkss\"" Nov 01 16:07:44.870018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.869987 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:44.910074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910041 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.910262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910120 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.910340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910291 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540387f-07a4-4601-b63a-99922263287f-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.910406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910383 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.910519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910506 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdxwm\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-kube-api-access-gdxwm\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.910564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910548 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.910679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910667 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.910716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910695 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.910801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.910786 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540387f-07a4-4601-b63a-99922263287f-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:44.960846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:44.960817 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:45.017534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.017382 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540387f-07a4-4601-b63a-99922263287f-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.017534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.017517 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.017840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.017643 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.017840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.017742 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540387f-07a4-4601-b63a-99922263287f-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.019089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.018706 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540387f-07a4-4601-b63a-99922263287f-istiod-ca-cert\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.019089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.018911 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.019089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.019088 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdxwm\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-kube-api-access-gdxwm\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.019358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.019151 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.019358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.019310 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.019358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.019353 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.029386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.022162 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-envoy\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.029386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.023588 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-certs\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.029386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.023841 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-data\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.029386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.024099 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.029386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.024333 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-credential-socket\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.082929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.073812 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540387f-07a4-4601-b63a-99922263287f-istio-podinfo\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.107771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.097587 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-istio-token\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.107771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.098311 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdxwm\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-kube-api-access-gdxwm\") pod \"e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.190789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.176251 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:45.319789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.318803 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:45.776647 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:07:45.776615 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf540387f_07a4_4601_b63a_99922263287f.slice/crio-07f97cd8f4d9cfe27c64848a9efe530bd3a87723917f3f75fd2cf1ade66f37ab WatchSource:0}: Error finding container 07f97cd8f4d9cfe27c64848a9efe530bd3a87723917f3f75fd2cf1ade66f37ab: Status 404 returned error can't find the container with id 07f97cd8f4d9cfe27c64848a9efe530bd3a87723917f3f75fd2cf1ade66f37ab Nov 01 16:07:45.777967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.777941 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6"] Nov 01 16:07:45.779872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.779844 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:45.780003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.779918 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:45.780003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:45.779954 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:07:46.042859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:46.041715 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:46.545776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:46.541947 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" event={"ID":"f540387f-07a4-4601-b63a-99922263287f","Type":"ContainerStarted","Data":"93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6"} Nov 01 16:07:46.545776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:46.542026 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" event={"ID":"f540387f-07a4-4601-b63a-99922263287f","Type":"ContainerStarted","Data":"07f97cd8f4d9cfe27c64848a9efe530bd3a87723917f3f75fd2cf1ade66f37ab"} Nov 01 16:07:46.567282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:46.567211 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podStartSLOduration=2.567192373 podStartE2EDuration="2.567192373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:46.562137351 +0000 UTC m=+1204.265303313" watchObservedRunningTime="2026-11-01 16:07:46.567192373 +0000 UTC m=+1204.270358310" Nov 01 16:07:47.034701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:47.034659 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:47.177314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:47.177281 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:47.184394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:47.184351 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:47.551536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:47.551503 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:47.552622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:47.552597 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:07:48.316169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:48.316124 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:49.115246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:49.115183 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:49.115741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:49.115722 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs"] Nov 01 16:07:49.173111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:49.173070 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf0cd681-stjrs"] Nov 01 16:07:49.248827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:49.248692 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:49.838330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:49.817546 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:49.963083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:49.962587 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:50.015951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.015640 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:50.047848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.027859 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:50.096190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.096060 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:50.097203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.097158 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:07:50.214449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.214353 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:50.352008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.351885 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:07:50.526311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.526190 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:50.569375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.569318 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:50.658268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.633377 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:50.807943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.801471 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63028bf5-b600-4781-9dbc-dde2cfb8917f" path="/var/lib/kubelet/pods/63028bf5-b600-4781-9dbc-dde2cfb8917f/volumes" Nov 01 16:07:50.963956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:50.962851 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.040778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.030811 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.040778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.031479 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:07:51.124899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.120887 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.124899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.121484 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:07:51.124899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.123215 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.124899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.123688 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:07:51.285953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.285843 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.480535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.480489 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:07:51.613110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.513921 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:51.613110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.589484 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:07:51.613110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.536464 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.613110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.590328 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:07:51.669381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.669331 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.692780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.683159 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:07:51.692780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.687864 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.754052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.706969 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:07:51.937570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.937444 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:51.976834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.966836 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:51.976834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:51.967449 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:07:52.059873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.038986 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.138282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.138228 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:07:52.169007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.168929 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.320829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.299117 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.320829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.318931 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:07:52.511778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.499896 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.633328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.617001 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.633328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.617448 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:07:52.702807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.685036 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:07:52.702807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.697877 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.823563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.776102 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:52.969830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:52.967846 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:53.057774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.053847 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:07:53.172789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.164877 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:07:53.347942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.173865 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:53.349818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.342323 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:07:53.408833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.377567 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:53.515956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.515894 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:53.524786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.516795 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:53.598912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.598815 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z"] Nov 01 16:07:53.642951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.636574 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:07:53.642951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.636992 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:07:53.706147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.696887 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:53.707025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.700884 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:53.836683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.836647 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:53.851913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.851829 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-f01de5\"" Nov 01 16:07:53.853328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:53.853302 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-f01de5-dockercfg-vsmtl\"" Nov 01 16:07:54.160775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.146147 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:54.191776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.166165 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z\" (UID: \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:54.192187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.192154 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4mj6\" (UniqueName: \"kubernetes.io/projected/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-kube-api-access-v4mj6\") pod \"payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z\" (UID: \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:54.343049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.343004 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.355258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.355212 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.375178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.362277 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:07:54.375178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.367777 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z\" (UID: \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:54.375178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.367900 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4mj6\" (UniqueName: \"kubernetes.io/projected/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-kube-api-access-v4mj6\") pod \"payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z\" (UID: \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:54.375178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.370091 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z\" (UID: \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:54.375178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.370218 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.473779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.468817 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:07:54.650045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.649937 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:54.650543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.650511 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:07:54.651152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:54.651126 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:07:54.798767 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:07:54.798710 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:07:55.137960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.137695 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:55.182942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.181505 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:07:55.225795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.185835 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:07:55.369318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.369276 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:07:55.379136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.379066 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:55.390827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.390716 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:55.421970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.421922 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:07:55.507775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.500199 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4mj6\" (UniqueName: \"kubernetes.io/projected/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-kube-api-access-v4mj6\") pod \"payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z\" (UID: \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:55.507775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.500273 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:55.648865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.648627 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:55.660007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.659950 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:55.661175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.661140 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:07:55.800111 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:07:55.791967 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.309s" Nov 01 16:07:55.800111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:55.792154 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z"] Nov 01 16:07:56.075744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.075707 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:56.256954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.244825 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:56.256954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.245158 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:07:56.256954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.245414 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Nov 01 16:07:56.387061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.377890 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:56.387061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.379827 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:56.420161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.408622 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:56.506841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.459833 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:56.527341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.514903 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:56.614044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.608651 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:56.684779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:56.674241 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:07:57.255901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.255858 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:07:57.317890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.317844 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:57.370501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.370460 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:57.715331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.399733 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:07:57.739085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.408938 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:57.741385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.430002 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:57.742021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.646818 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:07:57.750807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.675945 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:07:57.750807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.739432 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:07:57.750807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.742812 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:07:57.750807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.742920 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s"] Nov 01 16:07:57.750807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.742960 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:07:57.750807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.742979 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5bc25d5a-9m85s"] Nov 01 16:07:57.750807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.743000 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7"] Nov 01 16:07:57.750807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.743018 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3b46e218-bw9c7"] Nov 01 16:07:57.757841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.757809 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk"] Nov 01 16:07:57.855826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.854217 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:07:57.929813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:57.907027 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78"] Nov 01 16:07:58.200175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.191071 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:58.200175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.191785 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:07:58.208665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.208605 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bb22d5f-54glk"] Nov 01 16:07:58.338130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.322710 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:07:58.749791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.713334 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-708957cd-rnk78"] Nov 01 16:07:58.749791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.713532 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:07:58.749791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.713561 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" event={"ID":"e416ad18-b9a1-4e75-8d3a-db6cf4a20203","Type":"ContainerStarted","Data":"efac1e7ce603989cc60e891ae4616bc3386b8d864ad717551407c4c971bb8200"} Nov 01 16:07:58.749791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.713637 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:07:58.749791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.713665 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z"] Nov 01 16:07:58.832218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.832179 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d2067eb-c85b-4c6b-866f-339bcd750044" path="/var/lib/kubelet/pods/9d2067eb-c85b-4c6b-866f-339bcd750044/volumes" Nov 01 16:07:58.833501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.833376 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a526e313-2c7d-485e-82b4-91b1e19016b7" path="/var/lib/kubelet/pods/a526e313-2c7d-485e-82b4-91b1e19016b7/volumes" Nov 01 16:07:58.834210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.834186 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa52ad9-568f-438f-8f40-4748c2444159" path="/var/lib/kubelet/pods/cfa52ad9-568f-438f-8f40-4748c2444159/volumes" Nov 01 16:07:58.835060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.835028 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d797d9a4-54a5-479b-a186-8d8d21b99f58" path="/var/lib/kubelet/pods/d797d9a4-54a5-479b-a186-8d8d21b99f58/volumes" Nov 01 16:07:58.863011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:58.862944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:59.575316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:59.575280 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:07:59.739086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:59.739024 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" event={"ID":"e416ad18-b9a1-4e75-8d3a-db6cf4a20203","Type":"ContainerStarted","Data":"a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157"} Nov 01 16:07:59.739264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:59.739115 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:07:59.763931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:07:59.763855 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" podStartSLOduration=8.763837013 podStartE2EDuration="8.763837013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:07:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:07:59.754977509 +0000 UTC m=+1217.458143446" watchObservedRunningTime="2026-11-01 16:07:59.763837013 +0000 UTC m=+1217.467002952" Nov 01 16:08:00.380292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:00.380258 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:01.147443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:01.147406 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:02.069377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:02.069336 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b"] Nov 01 16:08:02.083664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:02.083629 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-690413d9-k2g2b"] Nov 01 16:08:02.135990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:02.135952 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:02.465455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:02.465404 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5" path="/var/lib/kubelet/pods/91fcbe13-b3b0-4c33-8a7e-e97c2aff7ac5/volumes" Nov 01 16:08:02.915877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:02.915840 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:03.660948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:03.660915 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:04.361587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:04.361558 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:05.082335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.082300 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:05.723870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.723838 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll"] Nov 01 16:08:05.728008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.727984 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.750740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.750695 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll"] Nov 01 16:08:05.838950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.838916 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.838950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.838957 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.839217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.838991 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.839217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.839021 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.839217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.839056 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.839217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.839107 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s68sq\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-kube-api-access-s68sq\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.839423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.839242 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.839423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.839384 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86900094-5677-4a05-b22e-f06a98775de6-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.839530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.839449 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86900094-5677-4a05-b22e-f06a98775de6-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.848680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.848641 4573 generic.go:356] "Generic (PLEG): container finished" podID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerID="2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89" exitCode=0 Nov 01 16:08:05.848839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.848717 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" event={"ID":"787148b3-4c00-43f9-b568-2cd1f994c3d9","Type":"ContainerDied","Data":"2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89"} Nov 01 16:08:05.849363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.849334 4573 scope.go:122] "RemoveContainer" containerID="2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89" Nov 01 16:08:05.851109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.851080 4573 generic.go:356] "Generic (PLEG): container finished" podID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerID="8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6" exitCode=0 Nov 01 16:08:05.851253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.851233 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" event={"ID":"30902ce9-0a12-45d5-a61a-bd13ed695adb","Type":"ContainerDied","Data":"8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6"} Nov 01 16:08:05.851589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.851511 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.851698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.851593 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.851698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.851635 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.851698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.851685 4573 scope.go:122] "RemoveContainer" containerID="8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6" Nov 01 16:08:05.852488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.852395 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:05.853842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.853809 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.853931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.853891 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.854001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.853936 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.854357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.854335 4573 generic.go:356] "Generic (PLEG): container finished" podID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerID="f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe" exitCode=0 Nov 01 16:08:05.854435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.854388 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" event={"ID":"84a241e7-c6e7-4a85-80d4-906fbde4253d","Type":"ContainerDied","Data":"f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe"} Nov 01 16:08:05.854818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.854802 4573 scope.go:122] "RemoveContainer" containerID="f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe" Nov 01 16:08:05.856420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.856390 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.856595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.856460 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.856595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.856501 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:05.957681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.957641 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.957899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.957692 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.957899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.957742 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.957899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.957793 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.957899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.957861 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.958157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.957923 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s68sq\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-kube-api-access-s68sq\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.958157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.958135 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.958412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.958367 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86900094-5677-4a05-b22e-f06a98775de6-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.958514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.958453 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86900094-5677-4a05-b22e-f06a98775de6-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.959303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.959276 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.962397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.960414 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.962397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.960643 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.964879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.964854 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.971283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.970335 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86900094-5677-4a05-b22e-f06a98775de6-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:05.993856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:05.992986 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:06.013807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.001719 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:06.039222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.033467 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s68sq\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-kube-api-access-s68sq\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:06.039222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.034102 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:06.039222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.035773 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86900094-5677-4a05-b22e-f06a98775de6-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:06.074224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.073951 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:06.098784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.096837 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:06.703405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.702844 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.703405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.702949 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.703405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.702994 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:06.790682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.785983 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll"] Nov 01 16:08:06.805648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.805566 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:06.993692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.973004 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" event={"ID":"84a241e7-c6e7-4a85-80d4-906fbde4253d","Type":"ContainerStarted","Data":"b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c"} Nov 01 16:08:06.993692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.982854 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" event={"ID":"86900094-5677-4a05-b22e-f06a98775de6","Type":"ContainerStarted","Data":"991a9d55bdb06ce0dcea1562ecf518b5f8a64f8848930f0a12c14044d60ac3ee"} Nov 01 16:08:06.993692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.985870 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" event={"ID":"787148b3-4c00-43f9-b568-2cd1f994c3d9","Type":"ContainerStarted","Data":"19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4"} Nov 01 16:08:06.993692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:06.990175 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" event={"ID":"30902ce9-0a12-45d5-a61a-bd13ed695adb","Type":"ContainerStarted","Data":"3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6"} Nov 01 16:08:07.259044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.258944 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:08:07.261075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.261046 4573 patch_prober.go:29] interesting pod/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:08:07.261286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.261261 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 16:08:07.341378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.341342 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:08:07.341731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.341707 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:07.341919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.341893 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:07.696774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.696699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:07.816667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.816633 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:08:07.817038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.817015 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 16:08:07.817346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.817317 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 16:08:07.853881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:07.853847 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:08:08.334981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.334943 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:08:08.385788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.375660 4573 patch_prober.go:29] interesting pod/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:08:08.385788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.375723 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 16:08:08.386272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.386241 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:08.386455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.386419 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:08.712880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.712832 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:08.753822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.744266 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:08.831780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.824415 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 16:08:08.831780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.824476 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 16:08:08.866223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.866087 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:08.940776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.937660 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" event={"ID":"86900094-5677-4a05-b22e-f06a98775de6","Type":"ContainerStarted","Data":"6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7"} Nov 01 16:08:08.980842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:08.976648 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:09.043184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.043147 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:09.180779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.159331 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:09.180779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.159899 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:08:09.180779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.159954 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 16:08:09.257135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.257004 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:09.258866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.258732 4573 patch_prober.go:29] interesting pod/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:08:09.259085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.259064 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 16:08:09.335861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.335799 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:09.337394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.337365 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/health\": context deadline exceeded" Nov 01 16:08:09.338106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.337992 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:09.338272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.338252 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:09.417023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.416989 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:08:09.533591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.533477 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:08:09.625367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.625320 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:09.647886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.647837 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:09.704918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.704871 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:08:09.714990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.714948 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:09.715861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.715833 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:08:09.756197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.756151 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:09.757185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.757163 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:08:09.780789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.764435 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podStartSLOduration=4.764415763 podStartE2EDuration="4.764415763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:09.011836204 +0000 UTC m=+1226.715002143" watchObservedRunningTime="2026-11-01 16:08:09.764415763 +0000 UTC m=+1227.467581704" Nov 01 16:08:09.822936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.822199 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 16:08:09.822936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.822263 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 16:08:09.992776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.987067 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:08:09.992776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:09.987571 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:08:10.073835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.028849 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": context deadline exceeded" Nov 01 16:08:10.182811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.146992 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:08:10.182811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.156802 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 16:08:10.182811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.174408 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:10.283371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.263297 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:08:10.283371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.264042 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:08:10.343910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.303540 4573 patch_prober.go:29] interesting pod/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:08:10.343910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.303603 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 16:08:10.343910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.337483 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:10.343910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.339144 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:08:10.343910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.343352 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:10.343910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.343394 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:10.586967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.586920 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:10.751959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.631066 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:10.777776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.673416 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:10.777776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.730311 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:10.777776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.770274 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:08:10.793654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.792820 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:08:10.825166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.822945 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 16:08:10.825166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.823004 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 16:08:10.868327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.852804 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:08:10.868765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.868729 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:08:10.871844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.871820 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj"] Nov 01 16:08:10.872873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.872843 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:08:10.873065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.873044 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:08:10.873207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.873196 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:08:10.968784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:10.949255 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264926e3-rg6jj"] Nov 01 16:08:11.038835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.038445 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:08:11.086997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.086939 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:11.106960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.106924 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:08:11.111982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.107355 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 16:08:11.111982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.108828 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:08:11.289101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.278963 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:11.342263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.338226 4573 patch_prober.go:29] interesting pod/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:08:11.342263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.338289 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 16:08:11.342263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.339516 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:11.342263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.339555 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:11.464804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.457711 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:08:11.503776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.492659 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:11.776403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.776357 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:11.795105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.795064 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:08:11.795731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.795697 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:08:11.796779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.796729 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:08:11.883775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.883552 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 16:08:11.883775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.883617 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 16:08:11.894771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:11.886590 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:12.053852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.051989 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:12.091783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.091730 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:08:12.092088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.092056 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 16:08:12.112250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.112206 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:08:12.189780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.170950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:12.269773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.267408 4573 patch_prober.go:29] interesting pod/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:08:12.269773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.267470 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 16:08:12.294770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.282162 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:12.355830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.351219 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:12.355830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.351284 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:12.564898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.564845 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:12.787875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.787810 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:08:12.789388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.789356 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:12.866814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.864527 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:12.897899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.895557 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:08:13.018173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:12.999024 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:13.026082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.020475 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 16:08:13.026082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.020547 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 16:08:13.068361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.068242 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:13.199778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.196684 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:08:13.229784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.226905 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:08:13.229784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.226993 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 16:08:13.288810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.265521 4573 patch_prober.go:29] interesting pod/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:08:13.288810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.265672 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 16:08:13.288810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.286870 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:08:13.356835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.348809 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:13.356835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.348880 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:13.472777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.444870 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9e4b7d-3fd8-4883-be91-e57d10e5e957" path="/var/lib/kubelet/pods/bc9e4b7d-3fd8-4883-be91-e57d10e5e957/volumes" Nov 01 16:08:13.549779 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:08:13.546714 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.097s" Nov 01 16:08:13.549779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.546830 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:08:13.574775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.574385 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:13.603515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.603474 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:13.675670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.675556 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8"] Nov 01 16:08:13.853777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.827653 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:13.853777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.836722 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:08:13.882879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.881887 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 16:08:13.882879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.881948 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 16:08:13.884417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.884385 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:13.921688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:13.921636 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:08:14.000324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.000223 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66"] Nov 01 16:08:14.060855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.060808 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-beb88848-2gvm8"] Nov 01 16:08:14.079440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.071877 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:14.080030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.079993 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:08:14.080279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.080248 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 16:08:14.102727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.102169 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:14.202369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.202328 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:08:14.203333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.203287 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:14.287937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.287836 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:14.479858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.341852 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2412c0c-xst66"] Nov 01 16:08:14.637449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.637404 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:14.638061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.638025 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:14.638331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.638310 4573 patch_prober.go:29] interesting pod/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 01 16:08:14.638467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.638448 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.832298 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.833941 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.834000 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.835133 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c949b85-1f70-447c-8016-c3f1d6d25714" path="/var/lib/kubelet/pods/3c949b85-1f70-447c-8016-c3f1d6d25714/volumes" Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.835981 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa44df9-6058-416f-a610-bf5ec65c41b5" path="/var/lib/kubelet/pods/afa44df9-6058-416f-a610-bf5ec65c41b5/volumes" Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.836291 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.837331 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.838914 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.843923 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Nov 01 16:08:14.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.845166 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:08:14.946848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.946809 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp"] Nov 01 16:08:14.971923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:14.971871 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:15.019208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.005620 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:15.019208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.010033 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.059040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.029183 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:08:15.059040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.041930 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp"] Nov 01 16:08:15.059040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.042185 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:08:15.060033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.059989 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:15.146829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.113579 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded" Nov 01 16:08:15.178872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.164012 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:08:15.231991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.182093 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8cq8\" (UniqueName: \"kubernetes.io/projected/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-kube-api-access-k8cq8\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.246667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.246633 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.247380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.247354 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.251019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.250995 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9"] Nov 01 16:08:15.253576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.253546 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:08:15.254309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.253647 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:08:15.254309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.254159 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 16:08:15.281050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.281018 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:08:15.308093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.308058 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:15.321102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.321071 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:08:15.334555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.333013 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:08:15.346676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.336420 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:08:15.349967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.349939 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.350239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.350208 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8cq8\" (UniqueName: \"kubernetes.io/projected/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-kube-api-access-k8cq8\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.350580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.350563 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.351153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.351122 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Nov 01 16:08:15.351280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.351181 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Nov 01 16:08:15.352790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.352773 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-758192c0-cxkg9"] Nov 01 16:08:15.352997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.352985 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:08:15.364923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.364879 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:08:15.379964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.379112 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:08:15.386887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.386862 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:08:15.405242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.405168 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.405463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.405361 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:08:15.424358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.424320 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.429803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.429781 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd"] Nov 01 16:08:15.439963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.439715 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8cq8\" (UniqueName: \"kubernetes.io/projected/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-kube-api-access-k8cq8\") pod \"maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.448832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.448805 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d2fb4-bnxxd"] Nov 01 16:08:15.510475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.510430 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:15.837201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.837121 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:08:15.843016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:15.842989 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:08:16.034557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.034507 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws"] Nov 01 16:08:16.043865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.043826 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79fb4e2d-6qkws"] Nov 01 16:08:16.064316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.064281 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:16.075414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.075386 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Nov 01 16:08:16.075548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.075443 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Nov 01 16:08:16.078236 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:08:16.078181 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742e34a1_2a9d_435a_b1ba_fcc8e6bb1d32.slice/crio-daa2fa3f2110d37cae0f9b97154f69b36296eaffd936f3a2bb2ad3ab2e918a2e WatchSource:0}: Error finding container daa2fa3f2110d37cae0f9b97154f69b36296eaffd936f3a2bb2ad3ab2e918a2e: Status 404 returned error can't find the container with id daa2fa3f2110d37cae0f9b97154f69b36296eaffd936f3a2bb2ad3ab2e918a2e Nov 01 16:08:16.079059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.079037 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp"] Nov 01 16:08:16.272556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.272519 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" event={"ID":"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32","Type":"ContainerStarted","Data":"daa2fa3f2110d37cae0f9b97154f69b36296eaffd936f3a2bb2ad3ab2e918a2e"} Nov 01 16:08:16.343466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.343419 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:08:16.344586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.344559 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:08:16.460924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.460890 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="006667a3-a418-4fea-abd2-6254d841716d" path="/var/lib/kubelet/pods/006667a3-a418-4fea-abd2-6254d841716d/volumes" Nov 01 16:08:16.461518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.461495 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453c59b1-3f5c-42b6-b61d-545c0a64b94c" path="/var/lib/kubelet/pods/453c59b1-3f5c-42b6-b61d-545c0a64b94c/volumes" Nov 01 16:08:16.462147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.462124 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fabc4f98-e053-40ce-9b8f-46c988a68604" path="/var/lib/kubelet/pods/fabc4f98-e053-40ce-9b8f-46c988a68604/volumes" Nov 01 16:08:16.834682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:16.834571 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:17.080501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:17.080469 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:17.081039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:17.080894 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:17.081823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:17.081738 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:08:17.284388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:17.284357 4573 generic.go:356] "Generic (PLEG): container finished" podID="742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32" containerID="87ddfdf5039620688e87742981ebecff2a933f044c680aff86ba9c3fd73c6ee3" exitCode=0 Nov 01 16:08:17.284573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:17.284428 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" event={"ID":"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32","Type":"ContainerDied","Data":"87ddfdf5039620688e87742981ebecff2a933f044c680aff86ba9c3fd73c6ee3"} Nov 01 16:08:17.464961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:17.464875 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:17.669103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:17.669072 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:18.487044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.487015 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:18.533737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.533693 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-kube-api-access-k8cq8\" (UniqueName: \"kubernetes.io/projected/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-kube-api-access-k8cq8\") pod \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " Nov 01 16:08:18.534069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.534043 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-xks-ca\") pod \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " Nov 01 16:08:18.534165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.534148 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-openshift-ca\") pod \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\" (UID: \"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32\") " Nov 01 16:08:18.534710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.534639 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-openshift-ca" pod "742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32" (UID: "742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:18.535632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.535609 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:18.536481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.536450 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-kube-api-access-k8cq8" pod "742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32" (UID: "742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32"). InnerVolumeSpecName "kube-api-access-k8cq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:18.536481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.536461 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-xks-ca" pod "742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32" (UID: "742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:18.540119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.540092 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:18.648773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.648207 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:18.648773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:18.648245 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8cq8\" (UniqueName: \"kubernetes.io/projected/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32-kube-api-access-k8cq8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:19.305309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:19.305277 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" Nov 01 16:08:19.305498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:19.305312 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp" event={"ID":"742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32","Type":"ContainerDied","Data":"daa2fa3f2110d37cae0f9b97154f69b36296eaffd936f3a2bb2ad3ab2e918a2e"} Nov 01 16:08:19.305498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:19.305366 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daa2fa3f2110d37cae0f9b97154f69b36296eaffd936f3a2bb2ad3ab2e918a2e" Nov 01 16:08:19.451828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:19.451729 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:20.165000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.164962 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:20.616410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.616375 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk"] Nov 01 16:08:20.617279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.617255 4573 cpu_manager.go:414] "containerMap: removing container" podUID="742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32" containerName="revoke-keys" Nov 01 16:08:20.617279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.617277 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32" containerName="revoke-keys" Nov 01 16:08:20.617490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.617476 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32" containerName="revoke-keys" Nov 01 16:08:20.621728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.621708 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.624608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.624586 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:08:20.625560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.625529 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk"] Nov 01 16:08:20.625682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.625581 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:08:20.689302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.689271 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.689888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.689850 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9pl2\" (UniqueName: \"kubernetes.io/projected/04059156-67c1-42ac-8c69-b0298c6c5c6f-kube-api-access-q9pl2\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.690347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.690320 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.751055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.751017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:20.792054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.792020 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.792268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.792087 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9pl2\" (UniqueName: \"kubernetes.io/projected/04059156-67c1-42ac-8c69-b0298c6c5c6f-kube-api-access-q9pl2\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.792268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.792226 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.792919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.792893 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.795042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.795015 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.801016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.800994 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9pl2\" (UniqueName: \"kubernetes.io/projected/04059156-67c1-42ac-8c69-b0298c6c5c6f-kube-api-access-q9pl2\") pod \"maas-api-revoke-sub-8626424e-9plkk\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.938691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.938607 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:20.944221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:20.944175 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:21.047579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:21.047539 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v"] Nov 01 16:08:21.058497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:21.058418 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dba73893-7lf9v"] Nov 01 16:08:21.283881 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:08:21.283850 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04059156_67c1_42ac_8c69_b0298c6c5c6f.slice/crio-dbf9badafac876ef6dd8b3fc357d758f74f6f896244123237bb135e11a1aa786 WatchSource:0}: Error finding container dbf9badafac876ef6dd8b3fc357d758f74f6f896244123237bb135e11a1aa786: Status 404 returned error can't find the container with id dbf9badafac876ef6dd8b3fc357d758f74f6f896244123237bb135e11a1aa786 Nov 01 16:08:21.285228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:21.285201 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk"] Nov 01 16:08:21.346994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:21.346962 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" event={"ID":"04059156-67c1-42ac-8c69-b0298c6c5c6f","Type":"ContainerStarted","Data":"dbf9badafac876ef6dd8b3fc357d758f74f6f896244123237bb135e11a1aa786"} Nov 01 16:08:21.664825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:21.664794 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:22.026509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.026433 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr"] Nov 01 16:08:22.032339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.032310 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl"] Nov 01 16:08:22.036927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.036899 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57077ffd-xstbr"] Nov 01 16:08:22.041608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.041578 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4b8258c-rt9tl"] Nov 01 16:08:22.359949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.359918 4573 generic.go:356] "Generic (PLEG): container finished" podID="04059156-67c1-42ac-8c69-b0298c6c5c6f" containerID="4d1d0ff6f7ce9bfe2f74d228394ee3d23190e3b6bd5ddfe19ca328e0805606fa" exitCode=0 Nov 01 16:08:22.360323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.360019 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" event={"ID":"04059156-67c1-42ac-8c69-b0298c6c5c6f","Type":"ContainerDied","Data":"4d1d0ff6f7ce9bfe2f74d228394ee3d23190e3b6bd5ddfe19ca328e0805606fa"} Nov 01 16:08:22.370886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.370849 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:22.532150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.532110 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea44c00-84b6-4cd8-9830-b20f1f5e4445" path="/var/lib/kubelet/pods/1ea44c00-84b6-4cd8-9830-b20f1f5e4445/volumes" Nov 01 16:08:22.534395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.533429 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5671ea17-8319-4db9-aff9-6854f88fb606" path="/var/lib/kubelet/pods/5671ea17-8319-4db9-aff9-6854f88fb606/volumes" Nov 01 16:08:22.534395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.534000 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d501b45a-9ba5-4db3-874f-bef00cf5b50b" path="/var/lib/kubelet/pods/d501b45a-9ba5-4db3-874f-bef00cf5b50b/volumes" Nov 01 16:08:22.648322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.648289 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv"] Nov 01 16:08:22.710709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.710672 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:22.811156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:22.811124 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:23.243788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.242129 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:23.643480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.643456 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:23.689453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.689425 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-openshift-ca\") pod \"04059156-67c1-42ac-8c69-b0298c6c5c6f\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " Nov 01 16:08:23.689759 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.689496 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04059156-67c1-42ac-8c69-b0298c6c5c6f-kube-api-access-q9pl2\" (UniqueName: \"kubernetes.io/projected/04059156-67c1-42ac-8c69-b0298c6c5c6f-kube-api-access-q9pl2\") pod \"04059156-67c1-42ac-8c69-b0298c6c5c6f\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " Nov 01 16:08:23.689759 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.689684 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-xks-ca\") pod \"04059156-67c1-42ac-8c69-b0298c6c5c6f\" (UID: \"04059156-67c1-42ac-8c69-b0298c6c5c6f\") " Nov 01 16:08:23.691126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.691098 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-openshift-ca" pod "04059156-67c1-42ac-8c69-b0298c6c5c6f" (UID: "04059156-67c1-42ac-8c69-b0298c6c5c6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:23.693159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.693130 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04059156-67c1-42ac-8c69-b0298c6c5c6f-kube-api-access-q9pl2" pod "04059156-67c1-42ac-8c69-b0298c6c5c6f" (UID: "04059156-67c1-42ac-8c69-b0298c6c5c6f"). InnerVolumeSpecName "kube-api-access-q9pl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:23.693609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.693581 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-xks-ca" pod "04059156-67c1-42ac-8c69-b0298c6c5c6f" (UID: "04059156-67c1-42ac-8c69-b0298c6c5c6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:23.792037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.792001 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:23.792037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.792031 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04059156-67c1-42ac-8c69-b0298c6c5c6f-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:23.792037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:23.792042 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9pl2\" (UniqueName: \"kubernetes.io/projected/04059156-67c1-42ac-8c69-b0298c6c5c6f-kube-api-access-q9pl2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:24.146445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:24.140141 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:24.368970 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:08:24.368919 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:08:24.403837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:24.397131 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" Nov 01 16:08:24.403837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:24.397160 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk" event={"ID":"04059156-67c1-42ac-8c69-b0298c6c5c6f","Type":"ContainerDied","Data":"dbf9badafac876ef6dd8b3fc357d758f74f6f896244123237bb135e11a1aa786"} Nov 01 16:08:24.403837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:24.397215 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbf9badafac876ef6dd8b3fc357d758f74f6f896244123237bb135e11a1aa786" Nov 01 16:08:24.421831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:24.421745 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:24.854866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:24.854782 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:25.027014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:25.026977 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk"] Nov 01 16:08:25.034252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:25.034225 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c809f72-nhkqk"] Nov 01 16:08:25.589609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:25.589578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:25.709778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:25.705835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:26.010275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.010233 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p"] Nov 01 16:08:26.011294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.011269 4573 cpu_manager.go:414] "containerMap: removing container" podUID="04059156-67c1-42ac-8c69-b0298c6c5c6f" containerName="revoke-keys" Nov 01 16:08:26.011294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.011291 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04059156-67c1-42ac-8c69-b0298c6c5c6f" containerName="revoke-keys" Nov 01 16:08:26.011494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.011481 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="04059156-67c1-42ac-8c69-b0298c6c5c6f" containerName="revoke-keys" Nov 01 16:08:26.015542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.015524 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026375 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istiod-ca-cert\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026434 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-podinfo\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026503 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2kq\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-kube-api-access-vs2kq\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026529 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-data\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026553 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-token\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026642 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026720 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-certs\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026883 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-envoy\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.026983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.026948 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-credential-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.031874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.029570 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5c6b75-openshift-default-dockercfg-j2qvf\"" Nov 01 16:08:26.047798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.042854 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p"] Nov 01 16:08:26.050437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.050368 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:26.057310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.057286 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8"] Nov 01 16:08:26.066093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.066067 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b9fb427-qcvp8"] Nov 01 16:08:26.176776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.174457 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istiod-ca-cert\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.176776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.174556 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-podinfo\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.176776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.174666 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2kq\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-kube-api-access-vs2kq\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.176776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.174699 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-data\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.176776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.174728 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-token\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.220028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.212293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:26.220028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.213420 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istiod-ca-cert\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.272419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.266657 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-podinfo\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.277253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.275256 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-data\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.277253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.275718 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.277253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.275840 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-certs\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.277253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.276024 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-envoy\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.277253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.276116 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-credential-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.277253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.276599 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-credential-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.277253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.276857 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-socket\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.277253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.277120 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-certs\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.334785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.330615 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-envoy\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.334785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.334728 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2kq\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-kube-api-access-vs2kq\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.347786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.342267 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-token\") pod \"e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.360789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.358084 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:26.584546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.584516 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:26.611276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.610950 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bc986a7-9816-4138-b597-5569cc44f31e" path="/var/lib/kubelet/pods/8bc986a7-9816-4138-b597-5569cc44f31e/volumes" Nov 01 16:08:26.612241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.611810 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f08d1fd8-46ba-4c0d-bcc3-270fe5582c81" path="/var/lib/kubelet/pods/f08d1fd8-46ba-4c0d-bcc3-270fe5582c81/volumes" Nov 01 16:08:26.972081 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:08:26.972049 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47cc643f_3c14_4ccb_bdec_cf64d5552a97.slice/crio-3c972e9bd5847b32a1ae63eddafaa6b3f38f062d43f1e87008f4a2fc66d2ea37 WatchSource:0}: Error finding container 3c972e9bd5847b32a1ae63eddafaa6b3f38f062d43f1e87008f4a2fc66d2ea37: Status 404 returned error can't find the container with id 3c972e9bd5847b32a1ae63eddafaa6b3f38f062d43f1e87008f4a2fc66d2ea37 Nov 01 16:08:26.972315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.972290 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p"] Nov 01 16:08:26.975440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.975237 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:26.975440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.975311 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:26.975440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:26.975354 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:27.033878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:27.033849 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664"] Nov 01 16:08:27.038933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:27.038908 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-357a04fb-8f664"] Nov 01 16:08:27.252037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:27.251955 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:27.598391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:27.598300 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" event={"ID":"47cc643f-3c14-4ccb-bdec-cf64d5552a97","Type":"ContainerStarted","Data":"5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762"} Nov 01 16:08:27.598391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:27.598354 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" event={"ID":"47cc643f-3c14-4ccb-bdec-cf64d5552a97","Type":"ContainerStarted","Data":"3c972e9bd5847b32a1ae63eddafaa6b3f38f062d43f1e87008f4a2fc66d2ea37"} Nov 01 16:08:27.619848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:27.619799 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" podStartSLOduration=2.619785007 podStartE2EDuration="2.619785007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:27.616788905 +0000 UTC m=+1245.319954833" watchObservedRunningTime="2026-11-01 16:08:27.619785007 +0000 UTC m=+1245.322950943" Nov 01 16:08:27.649577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:27.649540 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" podUID="a512c9e7-ddc2-4c13-93a3-6c828dbafce7" containerName="payload-processing" containerID="cri-o://9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c" gracePeriod=25 Nov 01 16:08:28.030994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.030962 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:08:28.032545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.032523 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f"] Nov 01 16:08:28.037627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.037603 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d351253-jq62f"] Nov 01 16:08:28.066965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.066924 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:28.148870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.145617 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-kube-api-access-ddnnf\" (UniqueName: \"kubernetes.io/projected/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-kube-api-access-ddnnf\") pod \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\" (UID: \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\") " Nov 01 16:08:28.148870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.145671 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-plugins-config-volume\") pod \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\" (UID: \"a512c9e7-ddc2-4c13-93a3-6c828dbafce7\") " Nov 01 16:08:28.148870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.147398 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-plugins-config-volume" pod "a512c9e7-ddc2-4c13-93a3-6c828dbafce7" (UID: "a512c9e7-ddc2-4c13-93a3-6c828dbafce7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:28.165774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.156219 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-kube-api-access-ddnnf" pod "a512c9e7-ddc2-4c13-93a3-6c828dbafce7" (UID: "a512c9e7-ddc2-4c13-93a3-6c828dbafce7"). InnerVolumeSpecName "kube-api-access-ddnnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:28.251080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.251026 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddnnf\" (UniqueName: \"kubernetes.io/projected/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-kube-api-access-ddnnf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:28.251080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.251064 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a512c9e7-ddc2-4c13-93a3-6c828dbafce7-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:28.377003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.376972 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:28.382921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.382896 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:28.460736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.460657 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5e8bd2-36db-4129-a074-83b8617ddfcf" path="/var/lib/kubelet/pods/3f5e8bd2-36db-4129-a074-83b8617ddfcf/volumes" Nov 01 16:08:28.461266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.461246 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775c8e45-c357-46f1-acc4-01adc262ff1d" path="/var/lib/kubelet/pods/775c8e45-c357-46f1-acc4-01adc262ff1d/volumes" Nov 01 16:08:28.608837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.608800 4573 generic.go:356] "Generic (PLEG): container finished" podID="a512c9e7-ddc2-4c13-93a3-6c828dbafce7" containerID="9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c" exitCode=0 Nov 01 16:08:28.609024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.608847 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" Nov 01 16:08:28.609024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.608886 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" event={"ID":"a512c9e7-ddc2-4c13-93a3-6c828dbafce7","Type":"ContainerDied","Data":"9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c"} Nov 01 16:08:28.609024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.608938 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv" event={"ID":"a512c9e7-ddc2-4c13-93a3-6c828dbafce7","Type":"ContainerDied","Data":"5c97d5cb4cb382ccdbec63acffa5253844dd969d1a852e2948d90b8d99138cf5"} Nov 01 16:08:28.609024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.608966 4573 scope.go:122] "RemoveContainer" containerID="9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c" Nov 01 16:08:28.609303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.609279 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:28.610465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.610445 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:28.621464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.621443 4573 scope.go:122] "RemoveContainer" containerID="9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c" Nov 01 16:08:28.621727 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:08:28.621707 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c\": container with ID starting with 9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c not found: ID does not exist" containerID="9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c" Nov 01 16:08:28.621828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.621739 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c"} err="failed to get container status \"9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c\": rpc error: code = NotFound desc = could not find container \"9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c\": container with ID starting with 9d57bc1690d53dc933a5e84470058a574769c78582aa8b4c952affad190b8d2c not found: ID does not exist" Nov 01 16:08:28.627671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.627651 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv"] Nov 01 16:08:28.634952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.634930 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-61bb54-579844cd75-zwdpv"] Nov 01 16:08:28.852270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:28.852196 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:29.175410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:29.175376 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:08:29.853985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:29.853943 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:29.957365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:29.957333 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:30.068382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:30.068341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:30.465699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:30.465661 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a512c9e7-ddc2-4c13-93a3-6c828dbafce7" path="/var/lib/kubelet/pods/a512c9e7-ddc2-4c13-93a3-6c828dbafce7/volumes" Nov 01 16:08:30.859578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:30.859537 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:30.967118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:30.967065 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:31.055708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:31.055674 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:31.173273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:31.173155 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:31.288342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:31.288303 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:31.983584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:31.983552 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:32.072848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:32.072808 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf"] Nov 01 16:08:32.091992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:32.091959 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb43ceb2-9nnxf"] Nov 01 16:08:32.460021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:32.459988 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ac7cd9-31bf-44d2-8242-992a00f08e56" path="/var/lib/kubelet/pods/a2ac7cd9-31bf-44d2-8242-992a00f08e56/volumes" Nov 01 16:08:32.985881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:32.981357 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:33.838000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:33.837949 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:34.597667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:34.597618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:35.261044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:35.261010 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:35.971271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:35.971235 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:36.688195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:36.688160 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:37.291566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:37.291532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:37.391578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:37.391544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:37.713911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:37.713851 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:37.812490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:37.812408 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:37.910873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:37.910841 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:38.296306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:38.296267 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:38.746207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:38.746172 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:38.846099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:38.846066 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:38.946704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:38.946664 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:39.072089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:39.072004 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:39.287343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:39.287309 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:39.580892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:39.580860 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:40.078278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.078223 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:40.760877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.760833 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:40.868907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.868865 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7"] Nov 01 16:08:40.870190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.870161 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a512c9e7-ddc2-4c13-93a3-6c828dbafce7" containerName="payload-processing" Nov 01 16:08:40.870373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.870356 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a512c9e7-ddc2-4c13-93a3-6c828dbafce7" containerName="payload-processing" Nov 01 16:08:40.870801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.870779 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a512c9e7-ddc2-4c13-93a3-6c828dbafce7" containerName="payload-processing" Nov 01 16:08:40.884828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.880969 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7"] Nov 01 16:08:40.884828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.881107 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:40.949303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.949265 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:40.949654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.949628 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsn7c\" (UniqueName: \"kubernetes.io/projected/2246fdf6-6d78-4217-a4cc-c9b36391c867-kube-api-access-jsn7c\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:40.949740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.949723 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:40.952808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.952241 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:08:40.952808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:40.952325 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:08:41.038040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.037960 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm"] Nov 01 16:08:41.045799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.045768 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bca74c7b-xv4qm"] Nov 01 16:08:41.051464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.051434 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:41.051732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.051711 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsn7c\" (UniqueName: \"kubernetes.io/projected/2246fdf6-6d78-4217-a4cc-c9b36391c867-kube-api-access-jsn7c\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:41.051845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.051790 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:41.052428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.052408 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:41.053654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.053631 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:41.060587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.060566 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsn7c\" (UniqueName: \"kubernetes.io/projected/2246fdf6-6d78-4217-a4cc-c9b36391c867-kube-api-access-jsn7c\") pod \"maas-api-revoke-sub-1139f235-cxsb7\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:41.270452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.270419 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:41.449327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.449281 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:41.597544 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:08:41.597495 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2246fdf6_6d78_4217_a4cc_c9b36391c867.slice/crio-1dab715394ae3356156238ff4e17a827c63069c9ee84f3c678faf318dcb03622 WatchSource:0}: Error finding container 1dab715394ae3356156238ff4e17a827c63069c9ee84f3c678faf318dcb03622: Status 404 returned error can't find the container with id 1dab715394ae3356156238ff4e17a827c63069c9ee84f3c678faf318dcb03622 Nov 01 16:08:41.602415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.602096 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7"] Nov 01 16:08:41.981206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.981175 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" event={"ID":"2246fdf6-6d78-4217-a4cc-c9b36391c867","Type":"ContainerStarted","Data":"1dab715394ae3356156238ff4e17a827c63069c9ee84f3c678faf318dcb03622"} Nov 01 16:08:41.983074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.983053 4573 generic.go:356] "Generic (PLEG): container finished" podID="f540387f-07a4-4601-b63a-99922263287f" containerID="93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6" exitCode=0 Nov 01 16:08:41.983163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.983124 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" event={"ID":"f540387f-07a4-4601-b63a-99922263287f","Type":"ContainerDied","Data":"93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6"} Nov 01 16:08:41.983450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.983436 4573 scope.go:122] "RemoveContainer" containerID="93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6" Nov 01 16:08:41.984959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.984919 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:41.985077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.985013 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:41.985077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:41.985063 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:42.011725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:42.011680 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" podStartSLOduration=2.011667364 podStartE2EDuration="2.011667364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:42.007263852 +0000 UTC m=+1259.710429792" watchObservedRunningTime="2026-11-01 16:08:42.011667364 +0000 UTC m=+1259.714833299" Nov 01 16:08:42.370681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:42.369633 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:42.465216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:42.465191 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895e104b-6d5d-4d63-88f8-055593234253" path="/var/lib/kubelet/pods/895e104b-6d5d-4d63-88f8-055593234253/volumes" Nov 01 16:08:42.979812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:42.975239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:43.002357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.002311 4573 generic.go:356] "Generic (PLEG): container finished" podID="2246fdf6-6d78-4217-a4cc-c9b36391c867" containerID="42ad5ff8dfae61a8a0e65f915fd8b1add4c7dc1778694f01af0b20b522ee0eb2" exitCode=0 Nov 01 16:08:43.002827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.002434 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" event={"ID":"2246fdf6-6d78-4217-a4cc-c9b36391c867","Type":"ContainerDied","Data":"42ad5ff8dfae61a8a0e65f915fd8b1add4c7dc1778694f01af0b20b522ee0eb2"} Nov 01 16:08:43.005362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.005300 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" event={"ID":"f540387f-07a4-4601-b63a-99922263287f","Type":"ContainerStarted","Data":"5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08"} Nov 01 16:08:43.215454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.214603 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:08:43.215677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.215569 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 16:08:43.215677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.215631 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 16:08:43.438948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.438913 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p"] Nov 01 16:08:43.439145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.439123 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" podUID="47cc643f-3c14-4ccb-bdec-cf64d5552a97" containerName="istio-proxy" containerID="cri-o://5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762" gracePeriod=30 Nov 01 16:08:43.612095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.612052 4573 patch_prober.go:29] interesting pod/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:08:43.612279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.612118 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" podUID="47cc643f-3c14-4ccb-bdec-cf64d5552a97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:08:43.696411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.696321 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:43.832904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.828869 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:43.873784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.873107 4573 scope.go:122] "RemoveContainer" containerID="8a2453b2e4a805cf8a8960e29faf9d1484b93f43d5683567a0debd1406b8d80e" Nov 01 16:08:43.971627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:43.971520 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:44.047525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.026265 4573 scope.go:122] "RemoveContainer" containerID="80fe4b551435e2623d308fc57bf2864362d46788f85e842094fd22ae9ca49d2d" Nov 01 16:08:44.116854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.113824 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578"] Nov 01 16:08:44.178571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.178535 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 16:08:44.178878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.178842 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 16:08:44.218434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.218398 4573 scope.go:122] "RemoveContainer" containerID="37bce1c9eafd62f7b38d097d3043116f60d3a9b1fcd810dd1c0810089646bec3" Nov 01 16:08:44.226015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.224045 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6961a50d-jm578"] Nov 01 16:08:44.423336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.423110 4573 scope.go:122] "RemoveContainer" containerID="6029b718d6f9535d488b974189b001de8987b27eaed3a42bccaa9200199641d9" Nov 01 16:08:44.423336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.423262 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp"] Nov 01 16:08:44.425020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.424858 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:44.437317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.436530 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-61bb54-lwgrp"] Nov 01 16:08:44.453282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.453263 4573 scope.go:122] "RemoveContainer" containerID="74904f30b3a95207e8a3ec79eae0ee8f82f302d92499113b6580d35914c291d9" Nov 01 16:08:44.675213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.673634 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32" path="/var/lib/kubelet/pods/742e34a1-2a9d-435a-b1ba-fcc8e6bb1d32/volumes" Nov 01 16:08:44.692993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.692509 4573 scope.go:122] "RemoveContainer" containerID="ac62b6aeeae299028831cff828400a2216cd300e94ab239d096cc7b9622d81a2" Nov 01 16:08:44.693158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.693041 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:44.707067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.698807 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8419a141-d63f-4bc4-b2ae-bbb424cb2e86" path="/var/lib/kubelet/pods/8419a141-d63f-4bc4-b2ae-bbb424cb2e86/volumes" Nov 01 16:08:44.808263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.808224 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:44.816782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.808743 4573 scope.go:122] "RemoveContainer" containerID="6f1a18ec2ab4bb7117b5798feef323c7f959e3733abe26ba9277ffca03c6a842" Nov 01 16:08:44.929004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.925248 4573 scope.go:122] "RemoveContainer" containerID="fe2bd43082c47013511a455ec4e0da388bc914112c33c723a852dca5db920a88" Nov 01 16:08:44.971743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:44.971707 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:45.035310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.035286 4573 scope.go:122] "RemoveContainer" containerID="ba80226fde011e6a4365fef0196ab2c062be783d4d4e4b4c53a63b705e5cf09e" Nov 01 16:08:45.249836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.235803 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2246fdf6-6d78-4217-a4cc-c9b36391c867-kube-api-access-jsn7c\" (UniqueName: \"kubernetes.io/projected/2246fdf6-6d78-4217-a4cc-c9b36391c867-kube-api-access-jsn7c\") pod \"2246fdf6-6d78-4217-a4cc-c9b36391c867\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " Nov 01 16:08:45.249836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.235904 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-openshift-ca\") pod \"2246fdf6-6d78-4217-a4cc-c9b36391c867\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " Nov 01 16:08:45.249836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.236217 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-xks-ca\") pod \"2246fdf6-6d78-4217-a4cc-c9b36391c867\" (UID: \"2246fdf6-6d78-4217-a4cc-c9b36391c867\") " Nov 01 16:08:45.249836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.236851 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 16:08:45.249836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.236906 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 16:08:45.283859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.262190 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2246fdf6-6d78-4217-a4cc-c9b36391c867-kube-api-access-jsn7c" pod "2246fdf6-6d78-4217-a4cc-c9b36391c867" (UID: "2246fdf6-6d78-4217-a4cc-c9b36391c867"). InnerVolumeSpecName "kube-api-access-jsn7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:45.283859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.262565 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-openshift-ca" pod "2246fdf6-6d78-4217-a4cc-c9b36391c867" (UID: "2246fdf6-6d78-4217-a4cc-c9b36391c867"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:45.348629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.348591 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-xks-ca" pod "2246fdf6-6d78-4217-a4cc-c9b36391c867" (UID: "2246fdf6-6d78-4217-a4cc-c9b36391c867"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:45.366581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.366538 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" event={"ID":"2246fdf6-6d78-4217-a4cc-c9b36391c867","Type":"ContainerDied","Data":"1dab715394ae3356156238ff4e17a827c63069c9ee84f3c678faf318dcb03622"} Nov 01 16:08:45.366814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.366793 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1dab715394ae3356156238ff4e17a827c63069c9ee84f3c678faf318dcb03622" Nov 01 16:08:45.367103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.367088 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7" Nov 01 16:08:45.372598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.372573 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsn7c\" (UniqueName: \"kubernetes.io/projected/2246fdf6-6d78-4217-a4cc-c9b36391c867-kube-api-access-jsn7c\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:45.373800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.373780 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:45.373927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.373913 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2246fdf6-6d78-4217-a4cc-c9b36391c867-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:45.497598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:45.497566 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:46.024285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:46.024245 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs"] Nov 01 16:08:46.028866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:46.028837 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2e2dfd-zmqgs"] Nov 01 16:08:46.187284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:46.187246 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 16:08:46.187582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:46.187549 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 16:08:46.287378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:46.287273 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:46.486778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:46.482960 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="227419a0-7ea0-40df-a1c1-68228db79e84" path="/var/lib/kubelet/pods/227419a0-7ea0-40df-a1c1-68228db79e84/volumes" Nov 01 16:08:47.004795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:47.004714 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:47.200809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:47.196582 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 16:08:47.205439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:47.203913 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 16:08:47.551915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:47.551879 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:08:47.685775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:47.685231 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:48.178291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.178255 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" start-of-body= Nov 01 16:08:48.178614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.178583 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.153:15021/healthz/ready\": dial tcp 10.133.0.153:15021: connect: connection refused" Nov 01 16:08:48.394811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.394772 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:48.837821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.837794 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:48.838185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838089 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istiod-ca-cert\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838155 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-certs\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838220 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-kube-api-access-vs2kq\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-kube-api-access-vs2kq\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838249 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-credential-socket\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838372 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-data\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838408 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-envoy\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838473 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-socket\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838494 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-certs" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:48.838600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838515 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istiod-ca-cert" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:48.838600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838511 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-podinfo\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838590 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-credential-socket" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:48.838879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838647 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-data" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:48.838879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838661 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-token\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-token\") pod \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\" (UID: \"47cc643f-3c14-4ccb-bdec-cf64d5552a97\") " Nov 01 16:08:48.838879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.838731 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-socket" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:48.839670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.839641 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:48.839805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.839675 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:48.839805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.839691 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:48.839805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.839707 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:48.839805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.839721 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:48.840527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.840503 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-podinfo" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:08:48.840617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.840608 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-kube-api-access-vs2kq" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "kube-api-access-vs2kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:48.840668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.840626 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-envoy" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:48.840825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.840809 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-token" pod "47cc643f-3c14-4ccb-bdec-cf64d5552a97" (UID: "47cc643f-3c14-4ccb-bdec-cf64d5552a97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:48.871190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.871159 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6"] Nov 01 16:08:48.872010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.871990 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2246fdf6-6d78-4217-a4cc-c9b36391c867" containerName="revoke-keys" Nov 01 16:08:48.872010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.872010 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2246fdf6-6d78-4217-a4cc-c9b36391c867" containerName="revoke-keys" Nov 01 16:08:48.872179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.872049 4573 cpu_manager.go:414] "containerMap: removing container" podUID="47cc643f-3c14-4ccb-bdec-cf64d5552a97" containerName="istio-proxy" Nov 01 16:08:48.872179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.872057 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cc643f-3c14-4ccb-bdec-cf64d5552a97" containerName="istio-proxy" Nov 01 16:08:48.872279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.872257 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cc643f-3c14-4ccb-bdec-cf64d5552a97" containerName="istio-proxy" Nov 01 16:08:48.872328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.872287 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2246fdf6-6d78-4217-a4cc-c9b36391c867" containerName="revoke-keys" Nov 01 16:08:48.876455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.876433 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:48.889910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.889067 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6"] Nov 01 16:08:48.941186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941141 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8xl\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-kube-api-access-rd8xl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:48.941383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941312 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:48.941454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941440 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:48.941524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941507 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:48.941626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941600 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:48.941775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941708 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:48.941775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941733 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:48.941918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941889 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs2kq\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-kube-api-access-vs2kq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:48.941918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941904 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:48.941918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941914 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:48.942081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:48.941923 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47cc643f-3c14-4ccb-bdec-cf64d5552a97-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:49.043124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.043092 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.043343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.043187 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.043343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.043249 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.043343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.043310 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.043540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.043386 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.043540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.043425 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.043660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.043585 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8xl\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-kube-api-access-rd8xl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.044123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.044093 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.045902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.045879 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.046160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.046135 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.046580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.046559 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.047015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.046998 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.051255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.051204 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8xl\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-kube-api-access-rd8xl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.051337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.051293 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fv7j6\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.157629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.157598 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:49.182157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.182129 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:08:49.183373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.183349 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:08:49.196409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.196387 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:49.641470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.641440 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7"] Nov 01 16:08:49.645080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.645050 4573 generic.go:356] "Generic (PLEG): container finished" podID="47cc643f-3c14-4ccb-bdec-cf64d5552a97" containerID="5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762" exitCode=0 Nov 01 16:08:49.648777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.645441 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" Nov 01 16:08:49.716545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.685140 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" event={"ID":"47cc643f-3c14-4ccb-bdec-cf64d5552a97","Type":"ContainerDied","Data":"5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762"} Nov 01 16:08:49.716545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.685184 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p" event={"ID":"47cc643f-3c14-4ccb-bdec-cf64d5552a97","Type":"ContainerDied","Data":"3c972e9bd5847b32a1ae63eddafaa6b3f38f062d43f1e87008f4a2fc66d2ea37"} Nov 01 16:08:49.716545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.685203 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7"] Nov 01 16:08:49.716545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.685241 4573 scope.go:122] "RemoveContainer" containerID="5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762" Nov 01 16:08:49.716545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.685470 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.716545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.690114 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:08:49.716545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.691119 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:08:49.728189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.728155 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.728408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.728390 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.728915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.728894 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvgg\" (UniqueName: \"kubernetes.io/projected/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-kube-api-access-vnvgg\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.757266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.757233 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p"] Nov 01 16:08:49.771414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.771058 4573 scope.go:122] "RemoveContainer" containerID="5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762" Nov 01 16:08:49.771799 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:08:49.771675 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762\": container with ID starting with 5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762 not found: ID does not exist" containerID="5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762" Nov 01 16:08:49.771799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.771715 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762"} err="failed to get container status \"5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762\": rpc error: code = NotFound desc = could not find container \"5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762\": container with ID starting with 5ddd6645b1abbaedb6a1e1e0dbbf13d06e0ee4ca0c28de28fba8848f26e30762 not found: ID does not exist" Nov 01 16:08:49.827771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.825224 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5c6b75-openshift-default-86f46b878f-cjk5p"] Nov 01 16:08:49.832592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.831724 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.855868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.850277 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.855868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.850453 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.855868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.850861 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvgg\" (UniqueName: \"kubernetes.io/projected/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-kube-api-access-vnvgg\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.855868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.855290 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.860071 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:08:49.856880 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d6c2dce_ecb9_4ca8_96a4_c2e0a80d6ca2.slice/crio-4b50589f3002dd76127f8a0be706e1379d596578779887a4370e43289b27d46f WatchSource:0}: Error finding container 4b50589f3002dd76127f8a0be706e1379d596578779887a4370e43289b27d46f: Status 404 returned error can't find the container with id 4b50589f3002dd76127f8a0be706e1379d596578779887a4370e43289b27d46f Nov 01 16:08:49.860928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.860598 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:49.865793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.865291 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:49.865936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.865860 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvgg\" (UniqueName: \"kubernetes.io/projected/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-kube-api-access-vnvgg\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-vx5l7\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:49.870453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:49.870431 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6"] Nov 01 16:08:50.019308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.019274 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:50.280333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.280231 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:50.330450 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:08:50.330402 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381853a5_4cde_403a_a0d2_a2eeacb1eaf8.slice/crio-369b4ed9e99cb0b008b13678b54bb67407f8e4256b06a48ab170e6a34a1d4646 WatchSource:0}: Error finding container 369b4ed9e99cb0b008b13678b54bb67407f8e4256b06a48ab170e6a34a1d4646: Status 404 returned error can't find the container with id 369b4ed9e99cb0b008b13678b54bb67407f8e4256b06a48ab170e6a34a1d4646 Nov 01 16:08:50.332814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.332791 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7"] Nov 01 16:08:50.492278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.492237 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cc643f-3c14-4ccb-bdec-cf64d5552a97" path="/var/lib/kubelet/pods/47cc643f-3c14-4ccb-bdec-cf64d5552a97/volumes" Nov 01 16:08:50.665023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.664614 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" event={"ID":"381853a5-4cde-403a-a0d2-a2eeacb1eaf8","Type":"ContainerStarted","Data":"369b4ed9e99cb0b008b13678b54bb67407f8e4256b06a48ab170e6a34a1d4646"} Nov 01 16:08:50.673418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.672364 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" event={"ID":"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2","Type":"ContainerStarted","Data":"8f7d405a247bc04c5c86ad7b9a23cd916c1aeb8ae078e4ff1a84cd53df32d4c8"} Nov 01 16:08:50.673418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.672425 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" event={"ID":"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2","Type":"ContainerStarted","Data":"4b50589f3002dd76127f8a0be706e1379d596578779887a4370e43289b27d46f"} Nov 01 16:08:50.673418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.672929 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:50.674999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.674969 4573 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fv7j6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:08:50.675138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.675028 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" podUID="3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:08:50.697085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:50.697032 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" podStartSLOduration=7.697017284 podStartE2EDuration="7.697017284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:50.691286929 +0000 UTC m=+1268.394452889" watchObservedRunningTime="2026-11-01 16:08:50.697017284 +0000 UTC m=+1268.400183219" Nov 01 16:08:51.191381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.191309 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:51.321771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.317504 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:51.527854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.527230 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:51.654446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.654407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:51.695017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.692521 4573 generic.go:356] "Generic (PLEG): container finished" podID="381853a5-4cde-403a-a0d2-a2eeacb1eaf8" containerID="9e2f7eee0515940777205d4100d83e2c96694af789b15da0219e21f11583a9eb" exitCode=0 Nov 01 16:08:51.695017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.694162 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" event={"ID":"381853a5-4cde-403a-a0d2-a2eeacb1eaf8","Type":"ContainerDied","Data":"9e2f7eee0515940777205d4100d83e2c96694af789b15da0219e21f11583a9eb"} Nov 01 16:08:51.695017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.694546 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:08:51.868119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.868081 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt"] Nov 01 16:08:51.868373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.868340 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" containerID="cri-o://3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6" gracePeriod=30 Nov 01 16:08:51.938984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:51.938950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:52.030514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.030480 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:52.174863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.174778 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:52.411773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.406802 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj"] Nov 01 16:08:52.436430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.435145 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.556278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.556235 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.556458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.556337 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6s8j\" (UniqueName: \"kubernetes.io/projected/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-kube-api-access-d6s8j\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.556458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.556450 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.603827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.595106 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj"] Nov 01 16:08:52.659333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.659276 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.659708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.659684 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.659802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.659784 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6s8j\" (UniqueName: \"kubernetes.io/projected/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-kube-api-access-d6s8j\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.660353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.660330 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.662816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.662792 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.668318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.668274 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6s8j\" (UniqueName: \"kubernetes.io/projected/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-kube-api-access-d6s8j\") pod \"maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.791627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.791591 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:52.906249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:52.906228 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:53.067677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.067591 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-kube-api-access-vnvgg\" (UniqueName: \"kubernetes.io/projected/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-kube-api-access-vnvgg\") pod \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " Nov 01 16:08:53.067858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.067683 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-xks-ca\") pod \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " Nov 01 16:08:53.067858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.067812 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-openshift-ca\") pod \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\" (UID: \"381853a5-4cde-403a-a0d2-a2eeacb1eaf8\") " Nov 01 16:08:53.068193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.068165 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-openshift-ca" pod "381853a5-4cde-403a-a0d2-a2eeacb1eaf8" (UID: "381853a5-4cde-403a-a0d2-a2eeacb1eaf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:53.068701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.068675 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:53.069495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.069473 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-kube-api-access-vnvgg" pod "381853a5-4cde-403a-a0d2-a2eeacb1eaf8" (UID: "381853a5-4cde-403a-a0d2-a2eeacb1eaf8"). InnerVolumeSpecName "kube-api-access-vnvgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:53.069791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.069734 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-xks-ca" pod "381853a5-4cde-403a-a0d2-a2eeacb1eaf8" (UID: "381853a5-4cde-403a-a0d2-a2eeacb1eaf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:53.096011 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:08:53.095985 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d48dce6_324e_4fe4_9c4b_96d1f2c165e5.slice/crio-143793fe8ecb684bc8ed595a59af343c1e9302df25ee0312867400216dbde7ec WatchSource:0}: Error finding container 143793fe8ecb684bc8ed595a59af343c1e9302df25ee0312867400216dbde7ec: Status 404 returned error can't find the container with id 143793fe8ecb684bc8ed595a59af343c1e9302df25ee0312867400216dbde7ec Nov 01 16:08:53.096672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.096642 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj"] Nov 01 16:08:53.170875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.170846 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:53.171037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.170892 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnvgg\" (UniqueName: \"kubernetes.io/projected/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-kube-api-access-vnvgg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:53.171037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.170913 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/381853a5-4cde-403a-a0d2-a2eeacb1eaf8-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:53.715226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.715155 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" event={"ID":"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5","Type":"ContainerStarted","Data":"936e23f79abd6dea8b1e5da10e4d0742c48e2d3d84226452dab3a6f12fff9d07"} Nov 01 16:08:53.715624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.715241 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" event={"ID":"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5","Type":"ContainerStarted","Data":"143793fe8ecb684bc8ed595a59af343c1e9302df25ee0312867400216dbde7ec"} Nov 01 16:08:53.716731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.716706 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" event={"ID":"381853a5-4cde-403a-a0d2-a2eeacb1eaf8","Type":"ContainerDied","Data":"369b4ed9e99cb0b008b13678b54bb67407f8e4256b06a48ab170e6a34a1d4646"} Nov 01 16:08:53.716858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.716793 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="369b4ed9e99cb0b008b13678b54bb67407f8e4256b06a48ab170e6a34a1d4646" Nov 01 16:08:53.716858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.716716 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7" Nov 01 16:08:53.733851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:53.733808 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" podStartSLOduration=1.733795152 podStartE2EDuration="1.733795152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:53.730440762 +0000 UTC m=+1271.433606697" watchObservedRunningTime="2026-11-01 16:08:53.733795152 +0000 UTC m=+1271.436961088" Nov 01 16:08:54.072915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:54.072887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:54.365541 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:08:54.365502 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:08:54.729203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:54.729115 4573 generic.go:356] "Generic (PLEG): container finished" podID="2d48dce6-324e-4fe4-9c4b-96d1f2c165e5" containerID="936e23f79abd6dea8b1e5da10e4d0742c48e2d3d84226452dab3a6f12fff9d07" exitCode=0 Nov 01 16:08:54.729203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:54.729166 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" event={"ID":"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5","Type":"ContainerDied","Data":"936e23f79abd6dea8b1e5da10e4d0742c48e2d3d84226452dab3a6f12fff9d07"} Nov 01 16:08:54.735604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:54.735579 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:55.582463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:55.582414 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:55.942534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:55.942513 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:56.064143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.064110 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-xks-ca\") pod \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " Nov 01 16:08:56.064330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.064174 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-openshift-ca\") pod \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " Nov 01 16:08:56.064457 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.064434 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-kube-api-access-d6s8j\" (UniqueName: \"kubernetes.io/projected/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-kube-api-access-d6s8j\") pod \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\" (UID: \"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5\") " Nov 01 16:08:56.064706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.064682 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-openshift-ca" pod "2d48dce6-324e-4fe4-9c4b-96d1f2c165e5" (UID: "2d48dce6-324e-4fe4-9c4b-96d1f2c165e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:56.065396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.065374 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:56.067206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.067176 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-kube-api-access-d6s8j" pod "2d48dce6-324e-4fe4-9c4b-96d1f2c165e5" (UID: "2d48dce6-324e-4fe4-9c4b-96d1f2c165e5"). InnerVolumeSpecName "kube-api-access-d6s8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:56.067401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.067371 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-xks-ca" pod "2d48dce6-324e-4fe4-9c4b-96d1f2c165e5" (UID: "2d48dce6-324e-4fe4-9c4b-96d1f2c165e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:08:56.166558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.166486 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6s8j\" (UniqueName: \"kubernetes.io/projected/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-kube-api-access-d6s8j\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:56.166558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.166524 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:56.275062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.275025 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:56.753585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.753511 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" Nov 01 16:08:56.753726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.753503 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj" event={"ID":"2d48dce6-324e-4fe4-9c4b-96d1f2c165e5","Type":"ContainerDied","Data":"143793fe8ecb684bc8ed595a59af343c1e9302df25ee0312867400216dbde7ec"} Nov 01 16:08:56.753726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:56.753639 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143793fe8ecb684bc8ed595a59af343c1e9302df25ee0312867400216dbde7ec" Nov 01 16:08:57.164669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.164637 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:57.179050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.179027 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:08:57.183219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183191 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-envoy\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.183333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183304 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-certs\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.183405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183349 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-credential-socket\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.183469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183420 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-podinfo\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.183469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183439 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-socket\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.183574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183472 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30902ce9-0a12-45d5-a61a-bd13ed695adb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30902ce9-0a12-45d5-a61a-bd13ed695adb-istiod-ca-cert\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.183574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183518 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-token\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-token\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.183670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183556 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-certs" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:57.183670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183638 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-credential-socket" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:57.183799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183737 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-socket" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:57.183982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.183949 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30902ce9-0a12-45d5-a61a-bd13ed695adb-istiod-ca-cert" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:08:57.184172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.184152 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-data\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.184274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.184257 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-kube-api-access-4n6nx\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-kube-api-access-4n6nx\") pod \"30902ce9-0a12-45d5-a61a-bd13ed695adb\" (UID: \"30902ce9-0a12-45d5-a61a-bd13ed695adb\") " Nov 01 16:08:57.184422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.184404 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-data" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:57.185146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.185121 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.185247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.185157 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.185247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.185174 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.185247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.185188 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.185247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.185203 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30902ce9-0a12-45d5-a61a-bd13ed695adb-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.185525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.185499 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-envoy" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:08:57.185915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.185893 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-token" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:57.186300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.186276 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-kube-api-access-4n6nx" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "kube-api-access-4n6nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:08:57.186880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.186858 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-podinfo" pod "30902ce9-0a12-45d5-a61a-bd13ed695adb" (UID: "30902ce9-0a12-45d5-a61a-bd13ed695adb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:08:57.215825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.215795 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc"] Nov 01 16:08:57.216727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.216707 4573 cpu_manager.go:414] "containerMap: removing container" podUID="381853a5-4cde-403a-a0d2-a2eeacb1eaf8" containerName="revoke-keys" Nov 01 16:08:57.216727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.216727 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381853a5-4cde-403a-a0d2-a2eeacb1eaf8" containerName="revoke-keys" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.216769 4573 cpu_manager.go:414] "containerMap: removing container" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.216777 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.216801 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2d48dce6-324e-4fe4-9c4b-96d1f2c165e5" containerName="revoke-keys" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.216808 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d48dce6-324e-4fe4-9c4b-96d1f2c165e5" containerName="revoke-keys" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.216838 4573 cpu_manager.go:414] "containerMap: removing container" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.216846 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.217053 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="381853a5-4cde-403a-a0d2-a2eeacb1eaf8" containerName="revoke-keys" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.217071 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.217084 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d48dce6-324e-4fe4-9c4b-96d1f2c165e5" containerName="revoke-keys" Nov 01 16:08:57.217233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.217100 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerName="istio-proxy" Nov 01 16:08:57.327715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.310931 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n6nx\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-kube-api-access-4n6nx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.327715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.310992 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.327715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.311016 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.327715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.311035 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30902ce9-0a12-45d5-a61a-bd13ed695adb-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:08:57.331569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.331520 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.333423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.333397 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc"] Nov 01 16:08:57.412497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.412465 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.412677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.412528 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.412677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.412618 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2thrl\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-kube-api-access-2thrl\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.412769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.412721 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.412873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.412857 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.412915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.412882 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.412986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.412972 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.413031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.413021 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.413155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.413144 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.500987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.500940 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:57.514279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514250 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514315 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514432 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514472 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514527 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514570 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2thrl\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-kube-api-access-2thrl\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514672 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514708 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-credential-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514708 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-socket\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514816 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.514915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514835 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.515079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.514940 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-data\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.515227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.515200 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-certs\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.515725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.515699 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istiod-ca-cert\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.517365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.517338 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-envoy\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.517454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.517347 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-podinfo\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.522087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.522063 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-token\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.522456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.522436 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2thrl\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-kube-api-access-2thrl\") pod \"e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.670839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.670807 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:57.770695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.770655 4573 generic.go:356] "Generic (PLEG): container finished" podID="30902ce9-0a12-45d5-a61a-bd13ed695adb" containerID="3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6" exitCode=0 Nov 01 16:08:57.770939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.770906 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" event={"ID":"30902ce9-0a12-45d5-a61a-bd13ed695adb","Type":"ContainerDied","Data":"3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6"} Nov 01 16:08:57.771073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.770975 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" event={"ID":"30902ce9-0a12-45d5-a61a-bd13ed695adb","Type":"ContainerDied","Data":"7bae41f8af375db7653a52dac594e96a4f789879cc4402092fdf6a9bb290341e"} Nov 01 16:08:57.771073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.771011 4573 scope.go:122] "RemoveContainer" containerID="3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6" Nov 01 16:08:57.771901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.771875 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt" Nov 01 16:08:57.797323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.796928 4573 scope.go:122] "RemoveContainer" containerID="8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6" Nov 01 16:08:57.812698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.812549 4573 scope.go:122] "RemoveContainer" containerID="3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6" Nov 01 16:08:57.813098 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:08:57.813051 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6\": container with ID starting with 3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6 not found: ID does not exist" containerID="3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6" Nov 01 16:08:57.813220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.813090 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6"} err="failed to get container status \"3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6\": rpc error: code = NotFound desc = could not find container \"3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6\": container with ID starting with 3df0ac0c0b3a7380455fad912c04573b75a917255ec0d8c56ada7e1420e39db6 not found: ID does not exist" Nov 01 16:08:57.813220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.813113 4573 scope.go:122] "RemoveContainer" containerID="8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6" Nov 01 16:08:57.813592 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:08:57.813570 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6\": container with ID starting with 8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6 not found: ID does not exist" containerID="8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6" Nov 01 16:08:57.813675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.813602 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6"} err="failed to get container status \"8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6\": rpc error: code = NotFound desc = could not find container \"8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6\": container with ID starting with 8065d5061845f0486766ac2e76ef9d84f8e0ce9c6d931e19efc963ecb1b5cbd6 not found: ID does not exist" Nov 01 16:08:57.817395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.817373 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt"] Nov 01 16:08:57.822014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.821991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:57.822377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.822360 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-61bb54-openshift-default-84944ff49c-p9zmt"] Nov 01 16:08:57.992675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.991459 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:57.992675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.991562 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:57.992675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.991606 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:08:58.005890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:57.999054 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc"] Nov 01 16:08:58.459450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:58.459408 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30902ce9-0a12-45d5-a61a-bd13ed695adb" path="/var/lib/kubelet/pods/30902ce9-0a12-45d5-a61a-bd13ed695adb/volumes" Nov 01 16:08:58.577560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:58.577521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:58.808440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:58.808360 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" event={"ID":"06c3e3b1-d636-46b7-bd80-4ac313ed0b07","Type":"ContainerStarted","Data":"56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60"} Nov 01 16:08:58.808624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:58.808611 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" event={"ID":"06c3e3b1-d636-46b7-bd80-4ac313ed0b07","Type":"ContainerStarted","Data":"33442464b7b9a1b44923a7a0a0ea5eee014aae635b264d68fd333c59ba59b890"} Nov 01 16:08:58.832161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:58.832103 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" podStartSLOduration=8.832083843 podStartE2EDuration="8.832083843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:08:58.830121823 +0000 UTC m=+1276.533287762" watchObservedRunningTime="2026-11-01 16:08:58.832083843 +0000 UTC m=+1276.535249780" Nov 01 16:08:59.263785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:59.263030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:59.693161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:59.689207 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:08:59.693826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:59.693802 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:08:59.694940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:59.694902 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 16:08:59.695031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:59.694954 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 16:08:59.793174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:08:59.793140 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:00.198147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:00.198111 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:00.683769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:00.683164 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 16:09:00.683769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:00.683259 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 16:09:01.130774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:01.122265 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:01.685786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:01.684880 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 16:09:01.685786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:01.684949 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 16:09:01.951166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:01.951083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:02.589780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:02.589096 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:02.682648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:02.675706 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 16:09:02.682648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:02.675793 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 16:09:02.687577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:02.687541 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:02.783600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:02.783544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:03.682795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:03.681902 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 16:09:03.682795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:03.681983 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 16:09:03.683371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:03.682856 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:04.385773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:04.383597 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:04.677364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:04.676683 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Nov 01 16:09:04.677364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:04.676794 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Nov 01 16:09:04.986132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:04.986031 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:05.289271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:05.289189 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:05.676629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:05.676598 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:09:05.677123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:05.677100 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:09:05.677956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:05.677939 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:09:05.764594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:05.764550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:05.895276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:05.895210 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:06.000922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:06.000662 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:06.346487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:06.346413 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:06.449613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:06.449557 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:06.574773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:06.574246 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:07.376187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:07.376142 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:08.053672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:08.053617 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:08.770516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:08.770484 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:09.465410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:09.465376 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:10.000703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.000669 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5"] Nov 01 16:09:10.006101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.006080 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.009836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.009813 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:09:10.010616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.010592 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:09:10.015467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.015442 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5"] Nov 01 16:09:10.039823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.039790 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.040170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.040144 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.040313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.040289 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk7jw\" (UniqueName: \"kubernetes.io/projected/59c031fe-ac69-4538-a859-ee38f0837757-kube-api-access-bk7jw\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.142083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.142048 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk7jw\" (UniqueName: \"kubernetes.io/projected/59c031fe-ac69-4538-a859-ee38f0837757-kube-api-access-bk7jw\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.142251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.142211 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.142485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.142468 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.143090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.143064 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.144500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.144480 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.150203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.150184 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk7jw\" (UniqueName: \"kubernetes.io/projected/59c031fe-ac69-4538-a859-ee38f0837757-kube-api-access-bk7jw\") pod \"maas-api-revoke-sub-1cfe49e0-26wg5\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.252345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.252266 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:10.334115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.334076 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:10.697052 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:09:10.697020 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c031fe_ac69_4538_a859_ee38f0837757.slice/crio-ec230566fd29c824477e49ef831399859aa861ad19094c6eea055e6e8f0aeaf4 WatchSource:0}: Error finding container ec230566fd29c824477e49ef831399859aa861ad19094c6eea055e6e8f0aeaf4: Status 404 returned error can't find the container with id ec230566fd29c824477e49ef831399859aa861ad19094c6eea055e6e8f0aeaf4 Nov 01 16:09:10.697268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.697246 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5"] Nov 01 16:09:10.949164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:10.949094 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:11.099244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:11.099174 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" event={"ID":"59c031fe-ac69-4538-a859-ee38f0837757","Type":"ContainerStarted","Data":"ec230566fd29c824477e49ef831399859aa861ad19094c6eea055e6e8f0aeaf4"} Nov 01 16:09:11.769026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:11.768991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:12.110463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:12.110425 4573 generic.go:356] "Generic (PLEG): container finished" podID="59c031fe-ac69-4538-a859-ee38f0837757" containerID="8276c377ff3e9808567f56efa4c2c9533bbcf70969d2ee7e77514f2a43e72515" exitCode=0 Nov 01 16:09:12.110919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:12.110514 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" event={"ID":"59c031fe-ac69-4538-a859-ee38f0837757","Type":"ContainerDied","Data":"8276c377ff3e9808567f56efa4c2c9533bbcf70969d2ee7e77514f2a43e72515"} Nov 01 16:09:12.459453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:12.459372 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:12.769048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:12.768974 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:13.296795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.296770 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:13.416158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.416076 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:13.475089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.458508 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-openshift-ca\") pod \"59c031fe-ac69-4538-a859-ee38f0837757\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " Nov 01 16:09:13.475693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.475658 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c031fe-ac69-4538-a859-ee38f0837757-kube-api-access-bk7jw\" (UniqueName: \"kubernetes.io/projected/59c031fe-ac69-4538-a859-ee38f0837757-kube-api-access-bk7jw\") pod \"59c031fe-ac69-4538-a859-ee38f0837757\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " Nov 01 16:09:13.479078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.479049 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-xks-ca\") pod \"59c031fe-ac69-4538-a859-ee38f0837757\" (UID: \"59c031fe-ac69-4538-a859-ee38f0837757\") " Nov 01 16:09:13.502829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.459916 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-openshift-ca" pod "59c031fe-ac69-4538-a859-ee38f0837757" (UID: "59c031fe-ac69-4538-a859-ee38f0837757"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:13.510124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.510083 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-xks-ca" pod "59c031fe-ac69-4538-a859-ee38f0837757" (UID: "59c031fe-ac69-4538-a859-ee38f0837757"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:09:13.520918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.511067 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c031fe-ac69-4538-a859-ee38f0837757-kube-api-access-bk7jw" pod "59c031fe-ac69-4538-a859-ee38f0837757" (UID: "59c031fe-ac69-4538-a859-ee38f0837757"). InnerVolumeSpecName "kube-api-access-bk7jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:13.581907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.581876 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:13.581907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.581902 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk7jw\" (UniqueName: \"kubernetes.io/projected/59c031fe-ac69-4538-a859-ee38f0837757-kube-api-access-bk7jw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:13.581907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:13.581914 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59c031fe-ac69-4538-a859-ee38f0837757-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:14.093625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:14.093551 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:14.130430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:14.130400 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" Nov 01 16:09:14.130605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:14.130428 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5" event={"ID":"59c031fe-ac69-4538-a859-ee38f0837757","Type":"ContainerDied","Data":"ec230566fd29c824477e49ef831399859aa861ad19094c6eea055e6e8f0aeaf4"} Nov 01 16:09:14.130605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:14.130477 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec230566fd29c824477e49ef831399859aa861ad19094c6eea055e6e8f0aeaf4" Nov 01 16:09:14.788504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:14.788465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:15.465064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:15.465018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:16.379904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:16.379839 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:17.178594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:17.178559 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:18.409018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:18.408977 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:19.761550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:19.761515 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:21.139556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:21.139525 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:22.093804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:22.093769 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:22.981673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:22.981635 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:23.961819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:23.961713 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:24.366780 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:09:24.366729 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:09:24.736742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:24.736658 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:25.619496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:25.619461 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:25.980904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:25.980808 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:26.080491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:26.080462 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:26.570149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:26.570119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:27.167344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:27.167312 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:27.267056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:27.267003 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:27.662666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:27.662626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:28.359979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:28.359944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:29.069138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:29.069102 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:29.847744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:29.847703 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:30.650011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:30.649980 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.048358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.048239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.240946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.240915 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.324835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.324694 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.426977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.426940 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.462894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.462863 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7"] Nov 01 16:09:31.467375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.467351 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-vx5l7"] Nov 01 16:09:31.526431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.526397 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.626162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.626127 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.748386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.748354 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:31.934614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.934533 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj"] Nov 01 16:09:31.952692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:31.952657 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-59d3a7-5dnjj"] Nov 01 16:09:32.217716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:32.217608 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:32.305980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:32.305935 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:32.470554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:32.470469 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d48dce6-324e-4fe4-9c4b-96d1f2c165e5" path="/var/lib/kubelet/pods/2d48dce6-324e-4fe4-9c4b-96d1f2c165e5/volumes" Nov 01 16:09:32.471220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:32.471201 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381853a5-4cde-403a-a0d2-a2eeacb1eaf8" path="/var/lib/kubelet/pods/381853a5-4cde-403a-a0d2-a2eeacb1eaf8/volumes" Nov 01 16:09:33.164920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:33.164887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:33.869076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:33.869041 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:34.771137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:34.771050 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:35.493887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:35.493849 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:35.670524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:35.670492 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:36.629994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:36.629902 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:37.464239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:37.464205 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:38.280525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:38.280493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:38.511790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:38.510217 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:38.608726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:38.608693 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:38.727804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:38.725660 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:38.827286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:38.827239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:38.926485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:38.926393 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:39.285884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:39.285739 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:39.683785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:39.682967 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:39.803120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:39.803067 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:40.554893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:40.554851 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:40.658048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:40.658013 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:41.412809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:41.412776 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:42.378651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:42.378611 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:42.486500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:42.483850 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:43.352196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:43.352166 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:44.306039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:44.306007 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:44.591416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:44.591279 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f"] Nov 01 16:09:44.591611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:44.591562 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" containerID="cri-o://a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4" gracePeriod=30 Nov 01 16:09:44.597931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:44.597901 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:44.819820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:44.819791 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:45.107570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.107522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:45.346337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.346306 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:45.527153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.527074 4573 scope.go:122] "RemoveContainer" containerID="127b4c15e8ce5f7b7cfd8bb8b2adc921430554562b884be33faf1dfc51a2460d" Nov 01 16:09:45.549315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.549254 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:45.552794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.552771 4573 scope.go:122] "RemoveContainer" containerID="47d202bb43e3b3b6903d0689ecf5c3919d3f06869f603df5a8fc2556f3658c87" Nov 01 16:09:45.563896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.563876 4573 scope.go:122] "RemoveContainer" containerID="0445caa5246c9ce5330c33c57e3887fff31655758990a087a6c60d9e426a9114" Nov 01 16:09:45.574327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.574310 4573 scope.go:122] "RemoveContainer" containerID="cea987321f075899d77879c1f53b46e4a79fc298bb2853d4f4967d0129b90480" Nov 01 16:09:45.584339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.584320 4573 scope.go:122] "RemoveContainer" containerID="c718751c833eb30717c1c20977852121227abeb027add8a1462f8cdc4e6de055" Nov 01 16:09:45.594445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.594277 4573 scope.go:122] "RemoveContainer" containerID="3acf874dfa957d7118e3e130058444f8ccebd8ad8e13f6fa9e75b7bd6cdbf304" Nov 01 16:09:45.604714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.604697 4573 scope.go:122] "RemoveContainer" containerID="1ccafdca17c2c8aa53072000aacbfdf35a408165edcf0cf13514bc566dcf0e38" Nov 01 16:09:45.614537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.614512 4573 scope.go:122] "RemoveContainer" containerID="83a496510e4836712cbf1afd6e053ae4e01cc12557db6b7c180b3356a00fa089" Nov 01 16:09:45.681765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.681583 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:45.809783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.808065 4573 scope.go:122] "RemoveContainer" containerID="ecf1a6a991e0be2f993d97225444f2868f73b3c2b106aa0e96539be4fc30290e" Nov 01 16:09:45.946990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:45.946824 4573 scope.go:122] "RemoveContainer" containerID="3e64c57bc5a7be75e8e84bf84041edf4df800702a2e2f6a457801f0f9a40b9ae" Nov 01 16:09:46.055068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.053477 4573 scope.go:122] "RemoveContainer" containerID="047123709b07131b916974dac91c190f175fb91d8a7b27cdc1cf5dd970904bd9" Nov 01 16:09:46.080065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.076487 4573 scope.go:122] "RemoveContainer" containerID="cf54da6f3816cb3ea902833ad2478289b14fa0287e98f872e806132845e6ef5f" Nov 01 16:09:46.107719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.107683 4573 scope.go:122] "RemoveContainer" containerID="8f680718532f5f1f1a5a1d9f45da50144cf8f828bb8886364b3ffa066e7867d5" Nov 01 16:09:46.119680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.119659 4573 scope.go:122] "RemoveContainer" containerID="babacd3c6b5798cddcaedb8fc0924001fdebe8c3abff58975b6da2c9593530c7" Nov 01 16:09:46.133250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.133228 4573 scope.go:122] "RemoveContainer" containerID="44855f3d0507f0342a457d544b814ca3e0f996229aa4dd56e0e2adc76fdebba3" Nov 01 16:09:46.147553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.147527 4573 scope.go:122] "RemoveContainer" containerID="ff6ec7dc44f7585845522966d418ce8beead9a3f32e228b155f280fc26b8e982" Nov 01 16:09:46.158701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.158680 4573 scope.go:122] "RemoveContainer" containerID="3adfef270fecfd4281ef0abe6ece49303d33f81f26ef04d7b27dd6505c1f9a3c" Nov 01 16:09:46.184388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.184347 4573 scope.go:122] "RemoveContainer" containerID="452f70ea7ad9b882c5d6347c6ecbbf2db84161676bf02b737c4be8c674651856" Nov 01 16:09:46.201466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.201441 4573 scope.go:122] "RemoveContainer" containerID="ed8941ff71d896eeeae2771f9b1108954284b9571bd0f7d476aea8652ccc999f" Nov 01 16:09:46.217146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.217122 4573 scope.go:122] "RemoveContainer" containerID="5774c3a21e1c595ba0207a17fbd923296b858da2027e0635129b5e33f27f6da1" Nov 01 16:09:46.229498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.228929 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:46.235965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.235939 4573 scope.go:122] "RemoveContainer" containerID="ee267ad91273fb140aaeeff62dd7a27befd2238e9be51c356be3f2a8096eb523" Nov 01 16:09:46.248286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.248265 4573 scope.go:122] "RemoveContainer" containerID="286ae588a1f17eb0327bacb67457682ff6bbb6537591f2381a1736937e562d65" Nov 01 16:09:46.258327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.258307 4573 scope.go:122] "RemoveContainer" containerID="09b6c465d5ad0aee494eebc6bd5b31bdb3c4dfd89fc9a7a52f652337143cc892" Nov 01 16:09:46.326796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.326734 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:46.598818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:46.594661 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:47.546648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:47.546617 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:48.161568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:48.161538 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:49.106013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.105978 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:49.421424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.421390 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:49.519178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.519148 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:49.907371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.907346 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:09:49.955078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955032 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-certs\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.955271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955141 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-token\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-token\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.955271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955183 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-data\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.955271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955246 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istiod-ca-cert\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.955449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955359 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-envoy\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.955660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955634 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-kube-api-access-4blpj\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-kube-api-access-4blpj\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.955795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955766 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-podinfo\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.955867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955824 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-credential-socket\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.955924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.955893 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-socket\") pod \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\" (UID: \"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8\") " Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.956176 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-data" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.956452 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-credential-socket" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.956518 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-certs" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.956594 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istiod-ca-cert" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.956728 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-socket" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.957161 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.957184 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.957200 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.957214 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:49.957251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.957229 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:49.958158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.958134 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-envoy" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:09:49.958491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.958468 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-token" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:49.959933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.959907 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-kube-api-access-4blpj" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "kube-api-access-4blpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:09:49.962999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:49.962874 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-podinfo" pod "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" (UID: "6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:09:50.058500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.058463 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.058500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.058495 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.058500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.058505 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.058762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.058514 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4blpj\" (UniqueName: \"kubernetes.io/projected/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8-kube-api-access-4blpj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:09:50.089295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.089227 4573 generic.go:356] "Generic (PLEG): container finished" podID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerID="a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4" exitCode=0 Nov 01 16:09:50.089295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.089280 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" Nov 01 16:09:50.089606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.089307 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" event={"ID":"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8","Type":"ContainerDied","Data":"a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4"} Nov 01 16:09:50.089606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.089394 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f" event={"ID":"6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8","Type":"ContainerDied","Data":"8aebe1f906861b96d011c661dd3753cd917ce0f0c1876849e745ff25fcccb654"} Nov 01 16:09:50.089606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.089418 4573 scope.go:122] "RemoveContainer" containerID="a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4" Nov 01 16:09:50.123054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.123027 4573 scope.go:122] "RemoveContainer" containerID="a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4" Nov 01 16:09:50.123411 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:09:50.123386 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4\": container with ID starting with a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4 not found: ID does not exist" containerID="a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4" Nov 01 16:09:50.123527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.123417 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4"} err="failed to get container status \"a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4\": rpc error: code = NotFound desc = could not find container \"a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4\": container with ID starting with a51be5c506ca77c6682a8e94338bb448e3ce43c07b223283f7119dc5370a05d4 not found: ID does not exist" Nov 01 16:09:50.150566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.150536 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f"] Nov 01 16:09:50.159574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.159523 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-qvf4f"] Nov 01 16:09:50.187739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.187705 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:50.283943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.283859 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:50.463689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.463655 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" path="/var/lib/kubelet/pods/6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8/volumes" Nov 01 16:09:50.630009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.627081 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:50.724700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:50.724667 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:51.546285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:51.546246 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:52.337361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:52.337333 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:53.098898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:53.090249 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:53.867968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:53.867930 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:54.365505 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:09:54.365472 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:09:54.562510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:54.562475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:55.274020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:55.273983 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:55.891870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:55.891842 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:56.581788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:56.581735 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:57.276255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:57.276217 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:57.933644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:57.933610 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:58.589919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:58.589886 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:59.490566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:59.490524 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8"] Nov 01 16:09:59.491994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:59.491958 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" containerID="cri-o://43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1" gracePeriod=30 Nov 01 16:09:59.639520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:59.639489 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:09:59.715971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:09:59.715915 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:00.008593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:00.008561 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:00.129035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:00.129001 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k"] Nov 01 16:10:00.129267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:00.129241 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" containerID="cri-o://b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88" gracePeriod=30 Nov 01 16:10:00.753775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:00.753728 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:01.359367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.359337 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:01.459714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.459673 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:01.563683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.563651 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:01.795025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.794944 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh"] Nov 01 16:10:01.796558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.796534 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" Nov 01 16:10:01.796692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.796680 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" Nov 01 16:10:01.796838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.796827 4573 cpu_manager.go:414] "containerMap: removing container" podUID="59c031fe-ac69-4538-a859-ee38f0837757" containerName="revoke-keys" Nov 01 16:10:01.796981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.796969 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c031fe-ac69-4538-a859-ee38f0837757" containerName="revoke-keys" Nov 01 16:10:01.797302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.797289 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c031fe-ac69-4538-a859-ee38f0837757" containerName="revoke-keys" Nov 01 16:10:01.797887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.797871 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f48c1b6-aaf2-4b54-a5d8-35d3b6b6a6a8" containerName="istio-proxy" Nov 01 16:10:01.802429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.802408 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.866882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.866848 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-166778-openshift-default-dockercfg-fctjj\"" Nov 01 16:10:01.871150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.871120 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh"] Nov 01 16:10:01.888057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888017 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6590e665-4fc6-42db-a481-775a1fd4d403-istiod-ca-cert\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.888238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888116 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-istio-token\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.888238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888194 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-envoy\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.888394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888373 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-data\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.888459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888423 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-credential-socket\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.888685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888658 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-socket\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.888743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888727 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-certs\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.888828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888793 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6590e665-4fc6-42db-a481-775a1fd4d403-istio-podinfo\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.888929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.888911 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv8lc\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-kube-api-access-mv8lc\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:01.912225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:01.912186 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:02.038061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.038024 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-envoy\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.051698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.051615 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-data\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.052382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.052360 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-credential-socket\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.052732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.052714 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-socket\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.053409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.053392 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-certs\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.053852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.053830 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6590e665-4fc6-42db-a481-775a1fd4d403-istio-podinfo\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.054061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.054044 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv8lc\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-kube-api-access-mv8lc\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.054251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.054235 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6590e665-4fc6-42db-a481-775a1fd4d403-istiod-ca-cert\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.054389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.054374 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-istio-token\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.067773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.053098 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-credential-socket\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.067773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.053307 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-socket\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.067773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.055431 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6590e665-4fc6-42db-a481-775a1fd4d403-istiod-ca-cert\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.067773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.052239 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-data\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.067773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.053742 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-certs\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.084487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.075990 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-envoy\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.111415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.111381 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6590e665-4fc6-42db-a481-775a1fd4d403-istio-podinfo\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.112362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.112336 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv8lc\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-kube-api-access-mv8lc\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.118879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.114863 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-istio-token\") pod \"e2e-api-a-166778-openshift-default-869f777c67-4ddgh\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.139383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.139349 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:02.198976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.186570 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:02.198976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.197704 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:10:02.198976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.197808 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:10:02.279824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.276322 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln"] Nov 01 16:10:02.303849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.293102 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.303849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.300965 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln"] Nov 01 16:10:02.317779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.309665 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:10:02.339143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.334532 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:10:02.418781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.413207 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.418781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.413445 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sltl\" (UniqueName: \"kubernetes.io/projected/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-kube-api-access-5sltl\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.418781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.413642 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.518378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.518334 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.518577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.518477 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.518708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.518688 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sltl\" (UniqueName: \"kubernetes.io/projected/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-kube-api-access-5sltl\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.520217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.519957 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.531841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.531365 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sltl\" (UniqueName: \"kubernetes.io/projected/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-kube-api-access-5sltl\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.532235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.532209 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.723580 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:10:02.723551 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6590e665_4fc6_42db_a481_775a1fd4d403.slice/crio-8f0ee02128968df3c38a7b1dd6ef99a54ed2c234c169d0de73ef849491bac5a5 WatchSource:0}: Error finding container 8f0ee02128968df3c38a7b1dd6ef99a54ed2c234c169d0de73ef849491bac5a5: Status 404 returned error can't find the container with id 8f0ee02128968df3c38a7b1dd6ef99a54ed2c234c169d0de73ef849491bac5a5 Nov 01 16:10:02.723798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.723773 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh"] Nov 01 16:10:02.726419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.726386 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:02.726491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.726451 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:02.726491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.726479 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:10:02.771802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.771776 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:02.990134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:02.990049 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:03.130859 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:10:03.125945 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbb85e6_a62f_4cfb_87f0_52dd50a34ff2.slice/crio-d0989d10c14812c6387c714d908e2697d155e4fe2250e8622dab69fe041e4574 WatchSource:0}: Error finding container d0989d10c14812c6387c714d908e2697d155e4fe2250e8622dab69fe041e4574: Status 404 returned error can't find the container with id d0989d10c14812c6387c714d908e2697d155e4fe2250e8622dab69fe041e4574 Nov 01 16:10:03.186986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:03.165680 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln"] Nov 01 16:10:03.493294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:03.493261 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" event={"ID":"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2","Type":"ContainerStarted","Data":"d0989d10c14812c6387c714d908e2697d155e4fe2250e8622dab69fe041e4574"} Nov 01 16:10:03.495017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:03.494989 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" event={"ID":"6590e665-4fc6-42db-a481-775a1fd4d403","Type":"ContainerStarted","Data":"56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81"} Nov 01 16:10:03.495140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:03.495035 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" event={"ID":"6590e665-4fc6-42db-a481-775a1fd4d403","Type":"ContainerStarted","Data":"8f0ee02128968df3c38a7b1dd6ef99a54ed2c234c169d0de73ef849491bac5a5"} Nov 01 16:10:03.524176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:03.524082 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podStartSLOduration=2.524067621 podStartE2EDuration="2.524067621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:03.517978718 +0000 UTC m=+1341.221144647" watchObservedRunningTime="2026-11-01 16:10:03.524067621 +0000 UTC m=+1341.227233556" Nov 01 16:10:03.791740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:03.791658 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:04.140281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.140251 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:04.146664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.146636 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:04.177354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.177316 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:10:04.656617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.650213 4573 generic.go:356] "Generic (PLEG): container finished" podID="9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2" containerID="c14c4ff006995e1b0a3a22fb352197f5fe758710e9bd56ca38622d23cbaa50b3" exitCode=0 Nov 01 16:10:04.656617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.652103 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" event={"ID":"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2","Type":"ContainerDied","Data":"c14c4ff006995e1b0a3a22fb352197f5fe758710e9bd56ca38622d23cbaa50b3"} Nov 01 16:10:04.656617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.652162 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:04.656617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.654270 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:10:04.933261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.933235 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:10:04.958272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958240 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-kube-api-access-qxt8d\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-kube-api-access-qxt8d\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.958430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958290 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-credential-socket\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.958430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958331 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-certs\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.958430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958394 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-envoy\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.958596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958546 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-socket\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.958810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958791 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d3e19be-9c70-4a83-8880-d4d5b4251407-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e19be-9c70-4a83-8880-d4d5b4251407-istiod-ca-cert\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.958895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958846 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-data\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.958948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958909 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-token\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.958948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958932 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-certs" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:04.959057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.958952 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-podinfo\") pod \"0d3e19be-9c70-4a83-8880-d4d5b4251407\" (UID: \"0d3e19be-9c70-4a83-8880-d4d5b4251407\") " Nov 01 16:10:04.959289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.959118 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-credential-socket" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:04.959401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.959379 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3e19be-9c70-4a83-8880-d4d5b4251407-istiod-ca-cert" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:04.959841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.959815 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e19be-9c70-4a83-8880-d4d5b4251407-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:04.959938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.959845 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:04.959938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.959860 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:04.960221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.960196 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-data" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:04.961605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.961561 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-socket" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:04.963041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.963013 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-kube-api-access-qxt8d" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "kube-api-access-qxt8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:04.963185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.963126 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-podinfo" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:04.963263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.963222 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-envoy" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:04.963661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.963641 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-token" pod "0d3e19be-9c70-4a83-8880-d4d5b4251407" (UID: "0d3e19be-9c70-4a83-8880-d4d5b4251407"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:04.966106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:04.966082 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:05.061878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.061846 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.061878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.061874 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.061878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.061883 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.062182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.061893 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxt8d\" (UniqueName: \"kubernetes.io/projected/0d3e19be-9c70-4a83-8880-d4d5b4251407-kube-api-access-qxt8d\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.062182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.061904 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.062182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.061914 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e19be-9c70-4a83-8880-d4d5b4251407-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.497412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.497382 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:10:05.574776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.574727 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-token\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-token\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.574965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.574834 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-data\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.574965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.574912 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-envoy\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.575083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575007 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-socket\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.575083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575054 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-credential-socket\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.575204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575105 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-certs\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.575204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575160 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-data" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.575204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575182 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-podinfo\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.575357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575326 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-kube-api-access-2pt42\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-kube-api-access-2pt42\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.575418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575371 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-socket" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.575418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575379 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istiod-ca-cert\") pod \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\" (UID: \"0018bf27-76e3-49c8-bec5-05cadb0f4a9a\") " Nov 01 16:10:05.576418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575670 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istiod-ca-cert" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.576418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575783 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-certs" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.576418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.575970 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-credential-socket" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.576418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.576321 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.576418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.576344 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.576418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.576362 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.576418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.576378 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.576418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.576393 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.578808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.578514 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-podinfo" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:05.578808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.578527 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-envoy" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:05.578808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.578560 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-token" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.578808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.578594 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-kube-api-access-2pt42" pod "0018bf27-76e3-49c8-bec5-05cadb0f4a9a" (UID: "0018bf27-76e3-49c8-bec5-05cadb0f4a9a"). InnerVolumeSpecName "kube-api-access-2pt42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.678581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.678540 4573 generic.go:356] "Generic (PLEG): container finished" podID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerID="b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88" exitCode=0 Nov 01 16:10:05.678942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.678661 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" event={"ID":"0018bf27-76e3-49c8-bec5-05cadb0f4a9a","Type":"ContainerDied","Data":"b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88"} Nov 01 16:10:05.678942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.678716 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" event={"ID":"0018bf27-76e3-49c8-bec5-05cadb0f4a9a","Type":"ContainerDied","Data":"ec5c9ed48451784c3c65033c8cee6b489e8af54cc123a4d1f42e88ac76439d1b"} Nov 01 16:10:05.678942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.678833 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.678942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.678857 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.678942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.678870 4573 scope.go:122] "RemoveContainer" containerID="b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88" Nov 01 16:10:05.679276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.678966 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k" Nov 01 16:10:05.679276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.678873 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.679276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.679056 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pt42\" (UniqueName: \"kubernetes.io/projected/0018bf27-76e3-49c8-bec5-05cadb0f4a9a-kube-api-access-2pt42\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:05.681176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.681154 4573 generic.go:356] "Generic (PLEG): container finished" podID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerID="43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1" exitCode=0 Nov 01 16:10:05.681272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.681195 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" Nov 01 16:10:05.681323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.681261 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" event={"ID":"0d3e19be-9c70-4a83-8880-d4d5b4251407","Type":"ContainerDied","Data":"43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1"} Nov 01 16:10:05.681592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.681569 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8" event={"ID":"0d3e19be-9c70-4a83-8880-d4d5b4251407","Type":"ContainerDied","Data":"54c1a8e62fc421d3c2ee4b17e7d9111038710eb770461db1da70814ba75c42cd"} Nov 01 16:10:05.698184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.698163 4573 scope.go:122] "RemoveContainer" containerID="b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88" Nov 01 16:10:05.698667 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:10:05.698645 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88\": container with ID starting with b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88 not found: ID does not exist" containerID="b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88" Nov 01 16:10:05.698738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.698675 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88"} err="failed to get container status \"b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88\": rpc error: code = NotFound desc = could not find container \"b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88\": container with ID starting with b0eecff49a90b268f795003f2baf37fbb8f2e4062005f5b99ce3f77d73035e88 not found: ID does not exist" Nov 01 16:10:05.698738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.698691 4573 scope.go:122] "RemoveContainer" containerID="43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1" Nov 01 16:10:05.709940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.709921 4573 scope.go:122] "RemoveContainer" containerID="43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1" Nov 01 16:10:05.710228 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:10:05.710209 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1\": container with ID starting with 43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1 not found: ID does not exist" containerID="43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1" Nov 01 16:10:05.710313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.710237 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1"} err="failed to get container status \"43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1\": rpc error: code = NotFound desc = could not find container \"43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1\": container with ID starting with 43d04101889bfe61170f958454c029f1c5deba8e4dd00ca3cf3cf69c71b9e6d1 not found: ID does not exist" Nov 01 16:10:05.741394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.741361 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k"] Nov 01 16:10:05.769040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.769011 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-lst2k"] Nov 01 16:10:05.814820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.814784 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8"] Nov 01 16:10:05.822316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.822289 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-ghbc8"] Nov 01 16:10:05.900466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.900438 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:05.964618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.964532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:05.984783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.983719 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-kube-api-access-5sltl\" (UniqueName: \"kubernetes.io/projected/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-kube-api-access-5sltl\") pod \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " Nov 01 16:10:05.984783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.984714 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-xks-ca\") pod \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " Nov 01 16:10:05.985427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.985398 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-openshift-ca\") pod \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\" (UID: \"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2\") " Nov 01 16:10:05.986837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.986788 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-openshift-ca" pod "9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2" (UID: "9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:05.989103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.989066 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-kube-api-access-5sltl" pod "9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2" (UID: "9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2"). InnerVolumeSpecName "kube-api-access-5sltl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:05.991805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:05.991574 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-xks-ca" pod "9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2" (UID: "9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:06.088323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.088291 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.088569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.088533 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sltl\" (UniqueName: \"kubernetes.io/projected/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-kube-api-access-5sltl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.088569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.088569 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:06.109646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.109615 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:06.228695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.228571 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:06.628662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.628040 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" path="/var/lib/kubelet/pods/0018bf27-76e3-49c8-bec5-05cadb0f4a9a/volumes" Nov 01 16:10:06.629224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.628674 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" path="/var/lib/kubelet/pods/0d3e19be-9c70-4a83-8880-d4d5b4251407/volumes" Nov 01 16:10:06.693511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.693478 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" event={"ID":"9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2","Type":"ContainerDied","Data":"d0989d10c14812c6387c714d908e2697d155e4fe2250e8622dab69fe041e4574"} Nov 01 16:10:06.693706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.693539 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0989d10c14812c6387c714d908e2697d155e4fe2250e8622dab69fe041e4574" Nov 01 16:10:06.693706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:06.693489 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln" Nov 01 16:10:07.254698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:07.254663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:07.371584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:07.371540 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:07.451044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:07.451012 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:08.045081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:08.045049 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:08.558003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:08.557961 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:09.455241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:09.455210 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:10.554081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:10.553929 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:11.344467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:11.344427 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:12.019639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:12.019607 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:12.446770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:12.446269 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:12.638222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:12.638168 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:12.854828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:12.853950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:13.546332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:13.546294 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:14.242582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:14.242550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:14.970437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:14.970404 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:15.760581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:15.760526 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:16.411880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:16.411846 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:17.181228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:17.181190 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:17.769246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:17.769211 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:18.468821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:18.468789 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:19.239298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.239265 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv"] Nov 01 16:10:19.264313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.264279 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2" containerName="revoke-keys" Nov 01 16:10:19.264536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.264519 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2" containerName="revoke-keys" Nov 01 16:10:19.264682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.264669 4573 cpu_manager.go:414] "containerMap: removing container" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" Nov 01 16:10:19.264798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.264784 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" Nov 01 16:10:19.264941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.264926 4573 cpu_manager.go:414] "containerMap: removing container" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" Nov 01 16:10:19.265056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.265044 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" Nov 01 16:10:19.265374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.265360 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="0018bf27-76e3-49c8-bec5-05cadb0f4a9a" containerName="istio-proxy" Nov 01 16:10:19.265480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.265470 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d3e19be-9c70-4a83-8880-d4d5b4251407" containerName="istio-proxy" Nov 01 16:10:19.265569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.265559 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2" containerName="revoke-keys" Nov 01 16:10:19.272137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.272111 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.301969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.291469 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:10:19.308772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.302235 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:10:19.355100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.355064 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv"] Nov 01 16:10:19.438161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.438112 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.438728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.438703 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxg8r\" (UniqueName: \"kubernetes.io/projected/5d9b2b0b-42ba-45fb-87f4-052118050d1c-kube-api-access-kxg8r\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.439044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.439026 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.458938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.458044 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:19.583490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.582018 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.583490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.582864 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxg8r\" (UniqueName: \"kubernetes.io/projected/5d9b2b0b-42ba-45fb-87f4-052118050d1c-kube-api-access-kxg8r\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.583490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.582996 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.599720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.586884 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.599720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.594092 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxg8r\" (UniqueName: \"kubernetes.io/projected/5d9b2b0b-42ba-45fb-87f4-052118050d1c-kube-api-access-kxg8r\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.599720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.594335 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-76063d10-f4pvv\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.655725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.655680 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:19.915663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.915624 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:19.948086 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:10:19.948048 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9b2b0b_42ba_45fb_87f4_052118050d1c.slice/crio-37fe646118eab1c77a1be53031b93be010cc569987a78b73bb471066624a2746 WatchSource:0}: Error finding container 37fe646118eab1c77a1be53031b93be010cc569987a78b73bb471066624a2746: Status 404 returned error can't find the container with id 37fe646118eab1c77a1be53031b93be010cc569987a78b73bb471066624a2746 Nov 01 16:10:19.948313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:19.948289 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv"] Nov 01 16:10:20.342505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:20.342476 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:20.446819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:20.444431 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:20.783045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:20.783011 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:20.891060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:20.891018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:20.930949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:20.927593 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" event={"ID":"5d9b2b0b-42ba-45fb-87f4-052118050d1c","Type":"ContainerStarted","Data":"93d326e82425694ffe090a3c02a71c8dfe755cc270b9f80ee8593e6dadbf7032"} Nov 01 16:10:20.930949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:20.927655 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" event={"ID":"5d9b2b0b-42ba-45fb-87f4-052118050d1c","Type":"ContainerStarted","Data":"37fe646118eab1c77a1be53031b93be010cc569987a78b73bb471066624a2746"} Nov 01 16:10:20.986775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:20.983099 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" podStartSLOduration=1.983067921 podStartE2EDuration="1.983067921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:20.960718784 +0000 UTC m=+1358.663884721" watchObservedRunningTime="2026-11-01 16:10:20.983067921 +0000 UTC m=+1358.686233858" Nov 01 16:10:21.471166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.471073 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:21.688661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.688628 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2"] Nov 01 16:10:21.693438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.693420 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.702347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.702312 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2"] Nov 01 16:10:21.835161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.835083 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.835348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.835324 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.835469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.835449 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptfm\" (UniqueName: \"kubernetes.io/projected/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-kube-api-access-hptfm\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.936708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.936672 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.936913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.936826 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptfm\" (UniqueName: \"kubernetes.io/projected/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-kube-api-access-hptfm\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.936994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.936912 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.937411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.937387 4573 generic.go:356] "Generic (PLEG): container finished" podID="5d9b2b0b-42ba-45fb-87f4-052118050d1c" containerID="93d326e82425694ffe090a3c02a71c8dfe755cc270b9f80ee8593e6dadbf7032" exitCode=0 Nov 01 16:10:21.937541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.937476 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" event={"ID":"5d9b2b0b-42ba-45fb-87f4-052118050d1c","Type":"ContainerDied","Data":"93d326e82425694ffe090a3c02a71c8dfe755cc270b9f80ee8593e6dadbf7032"} Nov 01 16:10:21.937729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.937701 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.939260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.939236 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:21.945155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:21.945131 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptfm\" (UniqueName: \"kubernetes.io/projected/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-kube-api-access-hptfm\") pod \"maas-api-revoke-sub-d98f52c9-v6nm2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:22.019309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:22.019281 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:22.247785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:22.247423 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:22.311373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:22.310581 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2"] Nov 01 16:10:22.968088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:22.967807 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" event={"ID":"74203b60-c4f9-4503-ac79-fcbccfb5a8f2","Type":"ContainerStarted","Data":"f03c5a81374fe9d7bc220a95baf1da06a2064a9134c3c712dce204b38999a949"} Nov 01 16:10:22.968088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:22.967891 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" event={"ID":"74203b60-c4f9-4503-ac79-fcbccfb5a8f2","Type":"ContainerStarted","Data":"04aefed9f7bcdf16f94cb638da50be15bc69d0b04f73da36d31ffe1071c95099"} Nov 01 16:10:23.015776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.010470 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:23.101100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.100953 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:23.186168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.186128 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:23.353696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.353435 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:23.381507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.381455 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" podStartSLOduration=2.3814394549999998 podStartE2EDuration="2.381439455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:10:23.011049105 +0000 UTC m=+1360.714215041" watchObservedRunningTime="2026-11-01 16:10:23.381439455 +0000 UTC m=+1361.084605392" Nov 01 16:10:23.425023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.424994 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-openshift-ca\") pod \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " Nov 01 16:10:23.425172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.425132 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9b2b0b-42ba-45fb-87f4-052118050d1c-kube-api-access-kxg8r\" (UniqueName: \"kubernetes.io/projected/5d9b2b0b-42ba-45fb-87f4-052118050d1c-kube-api-access-kxg8r\") pod \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " Nov 01 16:10:23.425317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.425285 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-xks-ca\") pod \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\" (UID: \"5d9b2b0b-42ba-45fb-87f4-052118050d1c\") " Nov 01 16:10:23.425457 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.425388 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-openshift-ca" pod "5d9b2b0b-42ba-45fb-87f4-052118050d1c" (UID: "5d9b2b0b-42ba-45fb-87f4-052118050d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:23.426379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.426349 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:23.427536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.427512 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9b2b0b-42ba-45fb-87f4-052118050d1c-kube-api-access-kxg8r" pod "5d9b2b0b-42ba-45fb-87f4-052118050d1c" (UID: "5d9b2b0b-42ba-45fb-87f4-052118050d1c"). InnerVolumeSpecName "kube-api-access-kxg8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:23.428884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.428858 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-xks-ca" pod "5d9b2b0b-42ba-45fb-87f4-052118050d1c" (UID: "5d9b2b0b-42ba-45fb-87f4-052118050d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:23.528403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.528370 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d9b2b0b-42ba-45fb-87f4-052118050d1c-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:23.528403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.528401 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxg8r\" (UniqueName: \"kubernetes.io/projected/5d9b2b0b-42ba-45fb-87f4-052118050d1c-kube-api-access-kxg8r\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:23.976407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.976377 4573 generic.go:356] "Generic (PLEG): container finished" podID="74203b60-c4f9-4503-ac79-fcbccfb5a8f2" containerID="f03c5a81374fe9d7bc220a95baf1da06a2064a9134c3c712dce204b38999a949" exitCode=0 Nov 01 16:10:23.976888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.976473 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" event={"ID":"74203b60-c4f9-4503-ac79-fcbccfb5a8f2","Type":"ContainerDied","Data":"f03c5a81374fe9d7bc220a95baf1da06a2064a9134c3c712dce204b38999a949"} Nov 01 16:10:23.979012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.978987 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" event={"ID":"5d9b2b0b-42ba-45fb-87f4-052118050d1c","Type":"ContainerDied","Data":"37fe646118eab1c77a1be53031b93be010cc569987a78b73bb471066624a2746"} Nov 01 16:10:23.979154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.979025 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv" Nov 01 16:10:23.979154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:23.979037 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37fe646118eab1c77a1be53031b93be010cc569987a78b73bb471066624a2746" Nov 01 16:10:24.147498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:24.147457 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:24.367321 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:10:24.367284 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:10:25.078662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.078630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:25.177793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.177671 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:25.301930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.293980 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:25.375775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.371342 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-xks-ca\") pod \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " Nov 01 16:10:25.375775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.371398 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-kube-api-access-hptfm\" (UniqueName: \"kubernetes.io/projected/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-kube-api-access-hptfm\") pod \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " Nov 01 16:10:25.375775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.371772 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-openshift-ca\") pod \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\" (UID: \"74203b60-c4f9-4503-ac79-fcbccfb5a8f2\") " Nov 01 16:10:25.416834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.410146 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-openshift-ca" pod "74203b60-c4f9-4503-ac79-fcbccfb5a8f2" (UID: "74203b60-c4f9-4503-ac79-fcbccfb5a8f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:25.416834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.412187 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-xks-ca" pod "74203b60-c4f9-4503-ac79-fcbccfb5a8f2" (UID: "74203b60-c4f9-4503-ac79-fcbccfb5a8f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:25.416834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.415139 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-kube-api-access-hptfm" pod "74203b60-c4f9-4503-ac79-fcbccfb5a8f2" (UID: "74203b60-c4f9-4503-ac79-fcbccfb5a8f2"). InnerVolumeSpecName "kube-api-access-hptfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:25.480990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.480954 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:25.481219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.481202 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:25.481349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:25.481335 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hptfm\" (UniqueName: \"kubernetes.io/projected/74203b60-c4f9-4503-ac79-fcbccfb5a8f2-kube-api-access-hptfm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:26.098843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:26.098800 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" event={"ID":"74203b60-c4f9-4503-ac79-fcbccfb5a8f2","Type":"ContainerDied","Data":"04aefed9f7bcdf16f94cb638da50be15bc69d0b04f73da36d31ffe1071c95099"} Nov 01 16:10:26.099343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:26.098866 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04aefed9f7bcdf16f94cb638da50be15bc69d0b04f73da36d31ffe1071c95099" Nov 01 16:10:26.099343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:26.098947 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2" Nov 01 16:10:26.148132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:26.148063 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:26.594185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:26.594143 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:26.697963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:26.697916 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:27.331098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:27.331070 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:28.097688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:28.092505 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:28.778379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:28.778340 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:29.365209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:29.365161 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:29.466060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:29.466026 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:29.568468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:29.568431 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:30.066038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:30.066003 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:30.452197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:30.452160 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:30.716605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:30.716523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:31.221155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:31.221119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:31.375812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:31.375777 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:31.557316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:31.557230 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:31.778487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:31.778454 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:32.043067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:32.043028 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:32.148788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:32.148720 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:32.886498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:32.886443 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:33.624828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:33.624784 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:34.268013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:34.267974 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:35.086814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:35.086743 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:35.502745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:35.502714 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:35.605064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:35.605019 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:35.959499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:35.959456 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:36.318108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:36.318017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:36.833031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:36.832988 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:36.964647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:36.964593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:37.176043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:37.175990 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:37.907639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:37.907603 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:38.654198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:38.654161 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:39.457734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:39.457673 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:40.652172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:40.652139 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:40.755476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:40.755411 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:40.854877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:40.854844 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:41.525914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:41.525883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:41.967022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:41.966982 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:42.076923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:42.076895 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:42.181551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:42.181514 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:42.411084 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:42.409989 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:42.744121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:42.743780 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:42.786001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:42.785930 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln"] Nov 01 16:10:42.844254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:42.844220 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5c6b75-sqrln"] Nov 01 16:10:42.874118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:42.874079 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:42.971718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:42.971607 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:43.066746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:43.066663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:43.229787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:43.225335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:43.896080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:43.896043 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:44.459891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:44.459861 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2" path="/var/lib/kubelet/pods/9fbb85e6-a62f-4cfb-87f0-52dd50a34ff2/volumes" Nov 01 16:10:44.568350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:44.568321 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:45.229523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:45.229490 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx"] Nov 01 16:10:45.229729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:45.229708 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" containerID="cri-o://cd891c59a5cc83970989bfa71b9eced23ec07f6af41fdb7d912f9a46c6b151ec" gracePeriod=30 Nov 01 16:10:45.324954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:45.324923 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:45.574018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:45.573935 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:10:45.574018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:45.574004 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:10:45.876805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:45.876719 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:46.060774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.051214 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:46.231933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.224450 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:46.395631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.395559 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv"] Nov 01 16:10:46.412874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.412827 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:46.417547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.417506 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-76063d10-f4pvv"] Nov 01 16:10:46.465337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.465300 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9b2b0b-42ba-45fb-87f4-052118050d1c" path="/var/lib/kubelet/pods/5d9b2b0b-42ba-45fb-87f4-052118050d1c/volumes" Nov 01 16:10:46.533718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.533618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:46.678253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.677282 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:46.965433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.965401 4573 scope.go:122] "RemoveContainer" containerID="d8f4410c99f1dff79940db2710a10bbf42023e115746fba5d98c3c21c4c0fa74" Nov 01 16:10:46.977273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.977254 4573 scope.go:122] "RemoveContainer" containerID="45f87fed7ebe19bea4624e1b501155466184d8a3b47bd498bdeaa0e2af60aab3" Nov 01 16:10:46.989739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:46.989720 4573 scope.go:122] "RemoveContainer" containerID="c2143545197cc383f3b11aa08fa63b40f996ea65f6dad89034f1c4f679e89253" Nov 01 16:10:47.009958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:47.009928 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:47.394993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:47.394959 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:47.487420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:47.487384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:47.640773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:47.621910 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:48.289374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:48.289285 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:48.953631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:48.953598 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:49.774523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:49.774450 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:50.061505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.061413 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:50.181776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.181655 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:50.386456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.386425 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:50.859929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.859896 4573 generic.go:356] "Generic (PLEG): container finished" podID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerID="cd891c59a5cc83970989bfa71b9eced23ec07f6af41fdb7d912f9a46c6b151ec" exitCode=0 Nov 01 16:10:50.861028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.859977 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" event={"ID":"5c44a69a-e1e5-4767-8915-204e17dc3461","Type":"ContainerDied","Data":"cd891c59a5cc83970989bfa71b9eced23ec07f6af41fdb7d912f9a46c6b151ec"} Nov 01 16:10:50.910330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.910302 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:10:50.998682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.998653 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr"] Nov 01 16:10:50.999379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999365 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5d9b2b0b-42ba-45fb-87f4-052118050d1c" containerName="revoke-keys" Nov 01 16:10:50.999427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999380 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9b2b0b-42ba-45fb-87f4-052118050d1c" containerName="revoke-keys" Nov 01 16:10:50.999427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999409 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" Nov 01 16:10:50.999427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999414 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" Nov 01 16:10:50.999542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999449 4573 cpu_manager.go:414] "containerMap: removing container" podUID="74203b60-c4f9-4503-ac79-fcbccfb5a8f2" containerName="revoke-keys" Nov 01 16:10:50.999542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999454 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74203b60-c4f9-4503-ac79-fcbccfb5a8f2" containerName="revoke-keys" Nov 01 16:10:50.999608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999559 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" containerName="istio-proxy" Nov 01 16:10:50.999608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999573 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="74203b60-c4f9-4503-ac79-fcbccfb5a8f2" containerName="revoke-keys" Nov 01 16:10:50.999608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:50.999582 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9b2b0b-42ba-45fb-87f4-052118050d1c" containerName="revoke-keys" Nov 01 16:10:51.003667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.003646 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.016205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.016181 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr"] Nov 01 16:10:51.023764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.023729 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-envoy\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.023903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.023826 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-data\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.023903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.023856 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-certs\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.023903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.023892 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-socket\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.024088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.024073 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-token\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-token\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.024201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.024147 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c44a69a-e1e5-4767-8915-204e17dc3461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c44a69a-e1e5-4767-8915-204e17dc3461-istiod-ca-cert\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.024314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.024300 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-credential-socket\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.024460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.024433 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-certs" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:51.024523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.024465 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-data" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:51.024523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.024472 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-kube-api-access-sdzxc\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-kube-api-access-sdzxc\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.024635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.024566 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-podinfo\") pod \"5c44a69a-e1e5-4767-8915-204e17dc3461\" (UID: \"5c44a69a-e1e5-4767-8915-204e17dc3461\") " Nov 01 16:10:51.026970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.026940 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c44a69a-e1e5-4767-8915-204e17dc3461-istiod-ca-cert" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:51.027350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.027328 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c44a69a-e1e5-4767-8915-204e17dc3461-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.027447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.027354 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.027447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.027369 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.027648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.027624 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-token" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:51.027736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.027635 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-credential-socket" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:51.027909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.027823 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-socket" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:51.029106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.029081 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-podinfo" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:10:51.029214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.029192 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-kube-api-access-sdzxc" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "kube-api-access-sdzxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:51.029521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.029491 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-envoy" pod "5c44a69a-e1e5-4767-8915-204e17dc3461" (UID: "5c44a69a-e1e5-4767-8915-204e17dc3461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:10:51.034279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.033467 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:10:51.034397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.034337 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:10:51.085042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.085010 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:51.146098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.132449 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.146098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.132799 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.146098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.132834 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2dj9\" (UniqueName: \"kubernetes.io/projected/b0061e80-728a-44c9-bb1f-1b09844b3f3a-kube-api-access-n2dj9\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.146098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.132992 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.146098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.133010 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.146098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.133024 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdzxc\" (UniqueName: \"kubernetes.io/projected/5c44a69a-e1e5-4767-8915-204e17dc3461-kube-api-access-sdzxc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.146098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.133039 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.172775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.148596 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.172775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.148635 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c44a69a-e1e5-4767-8915-204e17dc3461-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:51.212794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.212026 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:51.250165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.250128 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.250396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.250377 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.250496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.250403 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2dj9\" (UniqueName: \"kubernetes.io/projected/b0061e80-728a-44c9-bb1f-1b09844b3f3a-kube-api-access-n2dj9\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.250835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.250806 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.252823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.252803 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.260325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.260306 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2dj9\" (UniqueName: \"kubernetes.io/projected/b0061e80-728a-44c9-bb1f-1b09844b3f3a-kube-api-access-n2dj9\") pod \"maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.323428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.323389 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:51.388804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.388774 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:51.619582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.619544 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr"] Nov 01 16:10:51.633924 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:10:51.633895 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0061e80_728a_44c9_bb1f_1b09844b3f3a.slice/crio-c443db8d6b37f5eed97c22eaceb79a6df056b75bd9db8fc9d526dad15d33c048 WatchSource:0}: Error finding container c443db8d6b37f5eed97c22eaceb79a6df056b75bd9db8fc9d526dad15d33c048: Status 404 returned error can't find the container with id c443db8d6b37f5eed97c22eaceb79a6df056b75bd9db8fc9d526dad15d33c048 Nov 01 16:10:51.689362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.689331 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:51.868680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.868646 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" Nov 01 16:10:51.868680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.868655 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx" event={"ID":"5c44a69a-e1e5-4767-8915-204e17dc3461","Type":"ContainerDied","Data":"9a8fc261632d6ba0037ddc90e35b107db62bf2582c548b9e8bacf52b678b94df"} Nov 01 16:10:51.869262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.868718 4573 scope.go:122] "RemoveContainer" containerID="cd891c59a5cc83970989bfa71b9eced23ec07f6af41fdb7d912f9a46c6b151ec" Nov 01 16:10:51.870323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.870290 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" event={"ID":"b0061e80-728a-44c9-bb1f-1b09844b3f3a","Type":"ContainerStarted","Data":"c443db8d6b37f5eed97c22eaceb79a6df056b75bd9db8fc9d526dad15d33c048"} Nov 01 16:10:51.895512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.895483 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx"] Nov 01 16:10:51.900052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:51.900032 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-wsrkx"] Nov 01 16:10:52.153564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:52.153522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:52.467254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:52.467219 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:52.469497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:52.469387 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c44a69a-e1e5-4767-8915-204e17dc3461" path="/var/lib/kubelet/pods/5c44a69a-e1e5-4767-8915-204e17dc3461/volumes" Nov 01 16:10:52.561892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:52.561861 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:52.958167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:52.958132 4573 generic.go:356] "Generic (PLEG): container finished" podID="b0061e80-728a-44c9-bb1f-1b09844b3f3a" containerID="e74a89009a90ebbeccf9fe183e2b97d55d3bee5eb8b8b633c15c0391897800ec" exitCode=0 Nov 01 16:10:52.959576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:52.958875 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" event={"ID":"b0061e80-728a-44c9-bb1f-1b09844b3f3a","Type":"ContainerDied","Data":"e74a89009a90ebbeccf9fe183e2b97d55d3bee5eb8b8b633c15c0391897800ec"} Nov 01 16:10:53.078703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:53.078658 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:53.771243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:53.771149 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:54.157273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.157251 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:54.261796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.260386 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0061e80-728a-44c9-bb1f-1b09844b3f3a-kube-api-access-n2dj9\" (UniqueName: \"kubernetes.io/projected/b0061e80-728a-44c9-bb1f-1b09844b3f3a-kube-api-access-n2dj9\") pod \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " Nov 01 16:10:54.261796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.260666 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-openshift-ca\") pod \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " Nov 01 16:10:54.261796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.260728 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-xks-ca\") pod \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\" (UID: \"b0061e80-728a-44c9-bb1f-1b09844b3f3a\") " Nov 01 16:10:54.280845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.279626 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-openshift-ca" pod "b0061e80-728a-44c9-bb1f-1b09844b3f3a" (UID: "b0061e80-728a-44c9-bb1f-1b09844b3f3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:10:54.297901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.291027 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:54.297901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.291614 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-xks-ca" pod "b0061e80-728a-44c9-bb1f-1b09844b3f3a" (UID: "b0061e80-728a-44c9-bb1f-1b09844b3f3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:10:54.326514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.318364 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0061e80-728a-44c9-bb1f-1b09844b3f3a-kube-api-access-n2dj9" pod "b0061e80-728a-44c9-bb1f-1b09844b3f3a" (UID: "b0061e80-728a-44c9-bb1f-1b09844b3f3a"). InnerVolumeSpecName "kube-api-access-n2dj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:10:54.368065 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:10:54.368028 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:10:54.392623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.392594 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2dj9\" (UniqueName: \"kubernetes.io/projected/b0061e80-728a-44c9-bb1f-1b09844b3f3a-kube-api-access-n2dj9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:54.392623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.392619 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0061e80-728a-44c9-bb1f-1b09844b3f3a-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:10:54.557326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.557227 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:54.630439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.630403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:54.741570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.741522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:54.991101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.991059 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" event={"ID":"b0061e80-728a-44c9-bb1f-1b09844b3f3a","Type":"ContainerDied","Data":"c443db8d6b37f5eed97c22eaceb79a6df056b75bd9db8fc9d526dad15d33c048"} Nov 01 16:10:54.991278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.991126 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c443db8d6b37f5eed97c22eaceb79a6df056b75bd9db8fc9d526dad15d33c048" Nov 01 16:10:54.991278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:54.991197 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr" Nov 01 16:10:55.460020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:55.459984 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:56.297686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:56.290502 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:56.848813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:56.848779 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:10:57.082448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:57.082394 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:58.101036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:58.100963 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:10:58.950379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:10:58.950337 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:00.201253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:00.201215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:00.801776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:00.801426 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:00.809774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:00.809240 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:11:01.017776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.016859 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.051835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.043716 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.133779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.132863 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.223811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.205992 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.468191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.468150 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:01.469312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.469278 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:11:01.604487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.604443 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.639858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.639819 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:01.826830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.823083 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.909843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.880843 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:01.909843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.881450 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:11:01.909843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.881508 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:11:01.909843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:01.909384 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:02.019941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.019900 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:11:02.105057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.022669 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:02.115965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.044236 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:02.135791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.105739 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:11:02.135791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.129206 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:11:02.164333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.164291 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded" Nov 01 16:11:02.166798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.166771 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:11:02.254101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.254048 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:02.280811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.256371 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:11:02.282310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.281410 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:11:02.342861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.324460 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:11:02.861787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.845046 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:11:03.109777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.909102 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded" Nov 01 16:11:03.157396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.937150 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:11:03.157616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:02.937341 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded" Nov 01 16:11:03.157718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.112234 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:03.157839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.112498 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded" Nov 01 16:11:03.157929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.134100 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:03.158013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.137979 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Nov 01 16:11:03.281880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.281822 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:03.310105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.310067 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:03.491494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.491399 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:03.530360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.530330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:03.536979 4573 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.053s" Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.537076 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.537125 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.537175 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.537210 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.537240 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.537270 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.537299 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:11:03.541941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.537329 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:11:03.574804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:03.570487 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:11:04.347949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:04.347914 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:05.270978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:05.270936 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:05.840589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:05.840544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:06.020039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:06.019989 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:06.975684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:06.975650 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:07.677186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:07.677149 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:08.368788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:08.368730 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:09.173958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:09.173922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:09.845558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:09.845500 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:09.968478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:09.968444 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:10.808883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:10.808847 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:11.661510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:11.661474 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:11.758913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:11.758845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:11.867389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:11.867341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:12.014299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:12.014210 4573 generic.go:356] "Generic (PLEG): container finished" podID="6590e665-4fc6-42db-a481-775a1fd4d403" containerID="56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81" exitCode=0 Nov 01 16:11:12.014473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:12.014289 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" event={"ID":"6590e665-4fc6-42db-a481-775a1fd4d403","Type":"ContainerDied","Data":"56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81"} Nov 01 16:11:12.014852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:12.014833 4573 scope.go:122] "RemoveContainer" containerID="56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81" Nov 01 16:11:12.016980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:12.016946 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:12.017104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:12.017042 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:12.017104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:12.017093 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:12.564173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:12.564117 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:13.025839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:13.025800 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" event={"ID":"6590e665-4fc6-42db-a481-775a1fd4d403","Type":"ContainerStarted","Data":"3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024"} Nov 01 16:11:13.140690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:13.140655 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:11:13.141966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:13.141942 4573 patch_prober.go:29] interesting pod/e2e-api-a-166778-openshift-default-869f777c67-4ddgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:11:13.142044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:13.142006 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 16:11:13.433695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:13.433644 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:14.140610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.140576 4573 patch_prober.go:29] interesting pod/e2e-api-a-166778-openshift-default-869f777c67-4ddgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:11:14.141066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.140631 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 16:11:14.198923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.198891 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f"] Nov 01 16:11:14.200229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.200199 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b0061e80-728a-44c9-bb1f-1b09844b3f3a" containerName="revoke-keys" Nov 01 16:11:14.200229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.200228 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0061e80-728a-44c9-bb1f-1b09844b3f3a" containerName="revoke-keys" Nov 01 16:11:14.200505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.200486 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0061e80-728a-44c9-bb1f-1b09844b3f3a" containerName="revoke-keys" Nov 01 16:11:14.206328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.205584 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.235806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.230250 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f"] Nov 01 16:11:14.293175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293133 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cc464-bdff-46b2-a87a-70bc07951d15-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.293382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293239 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.293382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293276 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.293382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293375 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cc464-bdff-46b2-a87a-70bc07951d15-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.293579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293441 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtb6r\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-kube-api-access-xtb6r\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.293579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293494 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.293700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293606 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.293700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293663 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.293893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.293710 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.314977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.314942 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:14.400112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.400017 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.400799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.400767 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.401715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.401690 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cc464-bdff-46b2-a87a-70bc07951d15-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.406889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.406865 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtb6r\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-kube-api-access-xtb6r\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.412874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.412840 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.413622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.413602 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.415217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.406741 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cc464-bdff-46b2-a87a-70bc07951d15-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.415384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.414054 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.415547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.413421 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.415719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.415703 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.416216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.416104 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.416379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.416363 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.416898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.416735 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.445778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.432975 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cc464-bdff-46b2-a87a-70bc07951d15-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.460085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.454386 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cc464-bdff-46b2-a87a-70bc07951d15-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.491959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.490068 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.510172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.510118 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtb6r\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-kube-api-access-xtb6r\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.513306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.513271 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.548785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.548743 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:14.861198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.861171 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f"] Nov 01 16:11:14.861535 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:11:14.861513 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385cc464_bdff_46b2_a87a_70bc07951d15.slice/crio-d599f8af41da6ab57a28eac0bb7dd3dcc117f5ad0cf61fa4d18f937735d34af7 WatchSource:0}: Error finding container d599f8af41da6ab57a28eac0bb7dd3dcc117f5ad0cf61fa4d18f937735d34af7: Status 404 returned error can't find the container with id d599f8af41da6ab57a28eac0bb7dd3dcc117f5ad0cf61fa4d18f937735d34af7 Nov 01 16:11:14.889697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.889659 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:14.889858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.889733 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:14.889858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:14.889782 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:15.044992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.044955 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" event={"ID":"385cc464-bdff-46b2-a87a-70bc07951d15","Type":"ContainerStarted","Data":"c05dba7b8f340af41b3cfef122000000f8fbd6f50e20d679fec8c9d7e0432948"} Nov 01 16:11:15.045194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.045012 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" event={"ID":"385cc464-bdff-46b2-a87a-70bc07951d15","Type":"ContainerStarted","Data":"d599f8af41da6ab57a28eac0bb7dd3dcc117f5ad0cf61fa4d18f937735d34af7"} Nov 01 16:11:15.089774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.089296 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podStartSLOduration=1.089273989 podStartE2EDuration="1.089273989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:15.080872588 +0000 UTC m=+1412.784038548" watchObservedRunningTime="2026-11-01 16:11:15.089273989 +0000 UTC m=+1412.792439926" Nov 01 16:11:15.140417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.140328 4573 patch_prober.go:29] interesting pod/e2e-api-a-166778-openshift-default-869f777c67-4ddgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:11:15.140417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.140378 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 16:11:15.151576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.151547 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:15.549110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.549032 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:15.550211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.550192 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 16:11:15.550269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:15.550237 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 16:11:16.030592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:16.030551 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:16.147775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:16.146193 4573 patch_prober.go:29] interesting pod/e2e-api-a-166778-openshift-default-869f777c67-4ddgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:11:16.147775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:16.146260 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 16:11:16.549556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:16.549518 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 16:11:16.550118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:16.549584 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 16:11:17.157774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:17.157048 4573 patch_prober.go:29] interesting pod/e2e-api-a-166778-openshift-default-869f777c67-4ddgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:11:17.157774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:17.157105 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 16:11:17.509850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:17.509727 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:17.567131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:17.567097 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 16:11:17.567713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:17.567680 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 16:11:18.140233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:18.140201 4573 patch_prober.go:29] interesting pod/e2e-api-a-166778-openshift-default-869f777c67-4ddgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:11:18.140411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:18.140252 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 16:11:18.549688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:18.549599 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 16:11:18.549688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:18.549663 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 16:11:18.605423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:18.605358 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:19.140512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:19.140474 4573 patch_prober.go:29] interesting pod/e2e-api-a-166778-openshift-default-869f777c67-4ddgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Nov 01 16:11:19.140739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:19.140555 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Nov 01 16:11:19.550153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:19.550058 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 16:11:19.550153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:19.550112 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 16:11:19.652423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:19.652388 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:11:19.696971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:19.696938 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:20.146538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:20.146501 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:11:20.147733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:20.147713 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:11:20.549554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:20.549465 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 16:11:20.549554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:20.549528 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 16:11:20.705266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:20.705229 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:21.322841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:21.322796 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:21.549612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:21.549572 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body= Nov 01 16:11:21.549803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:21.549632 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" Nov 01 16:11:21.748373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:21.748338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:22.240950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.240911 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:22.555599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.555522 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:22.556086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.556063 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:22.556697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.556681 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:22.679979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.679942 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m"] Nov 01 16:11:22.687345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.687320 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.696600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.696181 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m"] Nov 01 16:11:22.701659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.700927 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:11:22.701659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.701614 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:11:22.741435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.741400 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:22.788979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.788928 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.789798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.789769 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.790102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.790083 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnmdn\" (UniqueName: \"kubernetes.io/projected/12db701c-3879-4426-a420-20cf4bc1632f-kube-api-access-mnmdn\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.892268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.892229 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.892445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.892339 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.892445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.892384 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnmdn\" (UniqueName: \"kubernetes.io/projected/12db701c-3879-4426-a420-20cf4bc1632f-kube-api-access-mnmdn\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.893399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.893369 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.896016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.895993 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:22.903899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:22.901567 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnmdn\" (UniqueName: \"kubernetes.io/projected/12db701c-3879-4426-a420-20cf4bc1632f-kube-api-access-mnmdn\") pod \"maas-api-revoke-keys-e2e-api-a-022f62-dpt9m\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:23.037710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:23.037681 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:23.341554 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:11:23.341520 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12db701c_3879_4426_a420_20cf4bc1632f.slice/crio-cb853ece7c2691e1df0eea6a81a13ca5a73a12e080a0e42441819fde573100d1 WatchSource:0}: Error finding container cb853ece7c2691e1df0eea6a81a13ca5a73a12e080a0e42441819fde573100d1: Status 404 returned error can't find the container with id cb853ece7c2691e1df0eea6a81a13ca5a73a12e080a0e42441819fde573100d1 Nov 01 16:11:23.341554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:23.341550 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m"] Nov 01 16:11:23.825602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:23.825065 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:24.188054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:24.188011 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" event={"ID":"12db701c-3879-4426-a420-20cf4bc1632f","Type":"ContainerStarted","Data":"238b2271370eb2e5702912b6fab09d60306172acdb2a58ca94b222cb74e85ac3"} Nov 01 16:11:24.188251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:24.188112 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" event={"ID":"12db701c-3879-4426-a420-20cf4bc1632f","Type":"ContainerStarted","Data":"cb853ece7c2691e1df0eea6a81a13ca5a73a12e080a0e42441819fde573100d1"} Nov 01 16:11:24.208639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:24.208586 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" podStartSLOduration=2.208572498 podStartE2EDuration="2.208572498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:24.203444267 +0000 UTC m=+1421.906610203" watchObservedRunningTime="2026-11-01 16:11:24.208572498 +0000 UTC m=+1421.911738473" Nov 01 16:11:24.365845 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:24.365805 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:11:24.781157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:24.778109 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:24.847469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:24.847439 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:24.945805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:24.945737 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:25.198731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:25.198695 4573 generic.go:356] "Generic (PLEG): container finished" podID="12db701c-3879-4426-a420-20cf4bc1632f" containerID="238b2271370eb2e5702912b6fab09d60306172acdb2a58ca94b222cb74e85ac3" exitCode=0 Nov 01 16:11:25.198968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:25.198783 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" event={"ID":"12db701c-3879-4426-a420-20cf4bc1632f","Type":"ContainerDied","Data":"238b2271370eb2e5702912b6fab09d60306172acdb2a58ca94b222cb74e85ac3"} Nov 01 16:11:25.723149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:25.723119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:26.404397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.404375 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:26.485884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.485798 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-openshift-ca\") pod \"12db701c-3879-4426-a420-20cf4bc1632f\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " Nov 01 16:11:26.485884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.485843 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12db701c-3879-4426-a420-20cf4bc1632f-kube-api-access-mnmdn\" (UniqueName: \"kubernetes.io/projected/12db701c-3879-4426-a420-20cf4bc1632f-kube-api-access-mnmdn\") pod \"12db701c-3879-4426-a420-20cf4bc1632f\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " Nov 01 16:11:26.486142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.485960 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-xks-ca\") pod \"12db701c-3879-4426-a420-20cf4bc1632f\" (UID: \"12db701c-3879-4426-a420-20cf4bc1632f\") " Nov 01 16:11:26.486307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.486277 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-openshift-ca" pod "12db701c-3879-4426-a420-20cf4bc1632f" (UID: "12db701c-3879-4426-a420-20cf4bc1632f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:26.486920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.486894 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:26.488097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.488072 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-xks-ca" pod "12db701c-3879-4426-a420-20cf4bc1632f" (UID: "12db701c-3879-4426-a420-20cf4bc1632f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:11:26.488494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.488479 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12db701c-3879-4426-a420-20cf4bc1632f-kube-api-access-mnmdn" pod "12db701c-3879-4426-a420-20cf4bc1632f" (UID: "12db701c-3879-4426-a420-20cf4bc1632f"). InnerVolumeSpecName "kube-api-access-mnmdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:26.556911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.556872 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:26.588869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.588842 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnmdn\" (UniqueName: \"kubernetes.io/projected/12db701c-3879-4426-a420-20cf4bc1632f-kube-api-access-mnmdn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:26.588869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.588866 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12db701c-3879-4426-a420-20cf4bc1632f-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:26.686370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.685421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:26.968729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:26.968697 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:27.229068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.228992 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" Nov 01 16:11:27.229068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.229031 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m" event={"ID":"12db701c-3879-4426-a420-20cf4bc1632f","Type":"ContainerDied","Data":"cb853ece7c2691e1df0eea6a81a13ca5a73a12e080a0e42441819fde573100d1"} Nov 01 16:11:27.229256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.229104 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb853ece7c2691e1df0eea6a81a13ca5a73a12e080a0e42441819fde573100d1" Nov 01 16:11:27.345255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.345213 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9"] Nov 01 16:11:27.346987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.346955 4573 cpu_manager.go:414] "containerMap: removing container" podUID="12db701c-3879-4426-a420-20cf4bc1632f" containerName="revoke-keys" Nov 01 16:11:27.346987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.346987 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12db701c-3879-4426-a420-20cf4bc1632f" containerName="revoke-keys" Nov 01 16:11:27.347188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.347172 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="12db701c-3879-4426-a420-20cf4bc1632f" containerName="revoke-keys" Nov 01 16:11:27.353352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.353322 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9"] Nov 01 16:11:27.353492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.353465 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.364433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.364266 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:27.389449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.389422 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2ce08a-openshift-default-dockercfg-x9s4x\"" Nov 01 16:11:27.400084 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400057 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-token\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.400192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400100 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2kl\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-kube-api-access-kd2kl\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.400249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400212 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-certs\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.400249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400234 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-podinfo\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.400429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400400 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3900c769-1183-45c0-aa46-0cb33ddc5778-istiod-ca-cert\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.400569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400551 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-data\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.400681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400664 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-credential-socket\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.400821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400806 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-envoy\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.400871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.400836 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-socket\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528144 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3900c769-1183-45c0-aa46-0cb33ddc5778-istiod-ca-cert\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528274 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-data\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528354 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-credential-socket\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528451 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-envoy\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528484 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-socket\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528601 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-token\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528638 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2kl\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-kube-api-access-kd2kl\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528708 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-certs\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.528732 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-podinfo\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.554206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.546433 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-socket\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.611774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.596520 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:27.652263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.650354 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-certs\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.652263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.650657 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3900c769-1183-45c0-aa46-0cb33ddc5778-istiod-ca-cert\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.763778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.762640 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:27.807844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.806943 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-data\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.807844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.807242 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-credential-socket\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.842773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.842501 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-token\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.854777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.854151 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-envoy\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.854777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.854673 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-podinfo\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.861793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.856732 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2kl\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-kube-api-access-kd2kl\") pod \"e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:27.986845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:27.986345 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:28.343813 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:11:28.343778 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3900c769_1183_45c0_aa46_0cb33ddc5778.slice/crio-859c62e960cd94989836bc9db3b43b86f842a4d5501b68ea7b4c3f902beff82f WatchSource:0}: Error finding container 859c62e960cd94989836bc9db3b43b86f842a4d5501b68ea7b4c3f902beff82f: Status 404 returned error can't find the container with id 859c62e960cd94989836bc9db3b43b86f842a4d5501b68ea7b4c3f902beff82f Nov 01 16:11:28.345907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:28.345884 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9"] Nov 01 16:11:28.347212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:28.347177 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:28.347325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:28.347261 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:28.347325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:28.347306 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:28.488884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:28.488847 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:28.590591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:28.590483 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:29.203169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.203139 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8"] Nov 01 16:11:29.208312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.208291 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.225638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.225607 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8"] Nov 01 16:11:29.250050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.249980 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.250214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.250103 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.250315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.250238 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1883148f-4863-492a-9da1-8787bc08c9c3-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.250488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.250464 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.250822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.250538 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.250822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.250566 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.250822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.250657 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7z7w\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-kube-api-access-t7z7w\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.250822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.250698 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1883148f-4863-492a-9da1-8787bc08c9c3-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.251161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.250854 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.253238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.253215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:29.269722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.269692 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" event={"ID":"3900c769-1183-45c0-aa46-0cb33ddc5778","Type":"ContainerStarted","Data":"61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33"} Nov 01 16:11:29.269870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.269746 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" event={"ID":"3900c769-1183-45c0-aa46-0cb33ddc5778","Type":"ContainerStarted","Data":"859c62e960cd94989836bc9db3b43b86f842a4d5501b68ea7b4c3f902beff82f"} Nov 01 16:11:29.296622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.296570 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podStartSLOduration=2.296553149 podStartE2EDuration="2.296553149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:29.293072801 +0000 UTC m=+1426.996238754" watchObservedRunningTime="2026-11-01 16:11:29.296553149 +0000 UTC m=+1426.999719087" Nov 01 16:11:29.352341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352313 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1883148f-4863-492a-9da1-8787bc08c9c3-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.352511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352450 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.352588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352515 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.352588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352532 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.352690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352603 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7z7w\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-kube-api-access-t7z7w\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.352690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352657 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1883148f-4863-492a-9da1-8787bc08c9c3-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.352841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352733 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.352914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352850 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.352914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.352899 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.353239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.353189 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-certs\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.353353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.353281 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.353572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.353549 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1883148f-4863-492a-9da1-8787bc08c9c3-istiod-ca-cert\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.353681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.353645 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-data\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.353735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.353692 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-credential-socket\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.355124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.355102 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1883148f-4863-492a-9da1-8787bc08c9c3-istio-podinfo\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.355256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.355236 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-envoy\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.365602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.365578 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7z7w\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-kube-api-access-t7z7w\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.370436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.370416 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-istio-token\") pod \"e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.529820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.529145 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:29.967227 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:11:29.967193 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1883148f_4863_492a_9da1_8787bc08c9c3.slice/crio-c86ba456551ab4f7245ff31d2e4591ef8b8097ad15a6fa4c91d55d90f05b3463 WatchSource:0}: Error finding container c86ba456551ab4f7245ff31d2e4591ef8b8097ad15a6fa4c91d55d90f05b3463: Status 404 returned error can't find the container with id c86ba456551ab4f7245ff31d2e4591ef8b8097ad15a6fa4c91d55d90f05b3463 Nov 01 16:11:29.967636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.967613 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8"] Nov 01 16:11:29.971144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.971110 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.971233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.971176 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.971233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.971209 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:29.988666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.988645 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:29.996404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:29.996384 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:30.282473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:30.282376 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" event={"ID":"1883148f-4863-492a-9da1-8787bc08c9c3","Type":"ContainerStarted","Data":"dabbc9eda559f6582cfdcf28be47e433c557ce80d517231ec47233a4ffc92283"} Nov 01 16:11:30.282473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:30.282442 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" event={"ID":"1883148f-4863-492a-9da1-8787bc08c9c3","Type":"ContainerStarted","Data":"c86ba456551ab4f7245ff31d2e4591ef8b8097ad15a6fa4c91d55d90f05b3463"} Nov 01 16:11:30.282927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:30.282893 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:30.283883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:30.283861 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:11:30.305197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:30.303961 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" podStartSLOduration=1.3039442270000001 podStartE2EDuration="1.303944227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:30.300742834 +0000 UTC m=+1428.003908771" watchObservedRunningTime="2026-11-01 16:11:30.303944227 +0000 UTC m=+1428.007110165" Nov 01 16:11:30.529655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:30.529614 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:30.649775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:30.649732 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:30.765788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:30.762826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:31.288362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:31.288313 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:31.531234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:31.531204 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:11:31.531440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:31.531259 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 16:11:31.619249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:31.619213 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:31.719701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:31.719664 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:31.851222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:31.851181 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:32.094783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:32.094657 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db"] Nov 01 16:11:32.095045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:32.095015 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" containerID="cri-o://0ea0c27334103b1a780f41702e4bd2c6c71d6a8ec791f7cc89a97eff1efb652b" gracePeriod=30 Nov 01 16:11:32.529824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:32.529791 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:11:32.530259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:32.529858 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 16:11:32.894860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:32.893684 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:32.994798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:32.994696 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:33.099791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:33.095261 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:33.533655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:33.532488 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:11:33.533655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:33.532575 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 16:11:33.816202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:33.816072 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:33.915394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:33.915361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:34.531161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:34.531127 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:11:34.531709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:34.531669 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 16:11:34.642742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:34.642711 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:35.283786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:35.283638 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:35.355006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:35.354969 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:35.491355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:35.491319 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:35.530407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:35.530372 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:11:35.530587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:35.530429 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 16:11:36.363890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:36.362386 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:36.448774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:36.446253 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:36.549613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:36.535274 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body= Nov 01 16:11:36.549613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:36.535369 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" Nov 01 16:11:36.808797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:36.808660 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:36.909496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:36.909467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:37.028823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.025429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:37.255634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.255598 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8"] Nov 01 16:11:37.255970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.255853 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" containerID="cri-o://dabbc9eda559f6582cfdcf28be47e433c557ce80d517231ec47233a4ffc92283" gracePeriod=30 Nov 01 16:11:37.298771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.298718 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m"] Nov 01 16:11:37.299231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.299197 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" containerID="cri-o://493ecf574aa1c08e7d0cc8d8c8fd873f8af785e11999a2670f2c9199c5897530" gracePeriod=30 Nov 01 16:11:37.314126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.314081 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:37.323111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.323069 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f"] Nov 01 16:11:37.323583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.323542 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" containerID="cri-o://c05dba7b8f340af41b3cfef122000000f8fbd6f50e20d679fec8c9d7e0432948" gracePeriod=30 Nov 01 16:11:37.688915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.688811 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:11:37.689576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.688910 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:11:37.729016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.725652 4573 generic.go:356] "Generic (PLEG): container finished" podID="e6235185-3f02-4ff5-b304-1fa951c09248" containerID="0ea0c27334103b1a780f41702e4bd2c6c71d6a8ec791f7cc89a97eff1efb652b" exitCode=0 Nov 01 16:11:37.729016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.725703 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" event={"ID":"e6235185-3f02-4ff5-b304-1fa951c09248","Type":"ContainerDied","Data":"0ea0c27334103b1a780f41702e4bd2c6c71d6a8ec791f7cc89a97eff1efb652b"} Nov 01 16:11:37.832514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.832484 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:11:37.911171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911143 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-kube-api-access-sr9sw\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-kube-api-access-sr9sw\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911203 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-socket\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911235 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-istio-token\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-istio-token\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911265 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-certs\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911314 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6235185-3f02-4ff5-b304-1fa951c09248-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6235185-3f02-4ff5-b304-1fa951c09248-istio-podinfo\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911383 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6235185-3f02-4ff5-b304-1fa951c09248-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6235185-3f02-4ff5-b304-1fa951c09248-istiod-ca-cert\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911450 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-data\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911577 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-credential-socket\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911635 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-envoy\") pod \"e6235185-3f02-4ff5-b304-1fa951c09248\" (UID: \"e6235185-3f02-4ff5-b304-1fa951c09248\") " Nov 01 16:11:37.911689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911572 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-certs" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:37.911689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911588 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-socket" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:37.911931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911862 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-data" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:37.911994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.911932 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6235185-3f02-4ff5-b304-1fa951c09248-istiod-ca-cert" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:37.912143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.912122 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-credential-socket" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:37.912434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.912415 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:37.912519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.912436 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:37.912519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.912446 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:37.912519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.912455 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:37.912519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.912464 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6235185-3f02-4ff5-b304-1fa951c09248-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:37.913594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.913573 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-kube-api-access-sr9sw" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "kube-api-access-sr9sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:37.913728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.913647 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-istio-token" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:37.914036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.913987 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6235185-3f02-4ff5-b304-1fa951c09248-istio-podinfo" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:11:37.914107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:37.914051 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-envoy" pod "e6235185-3f02-4ff5-b304-1fa951c09248" (UID: "e6235185-3f02-4ff5-b304-1fa951c09248"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:38.013952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.013874 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:38.013952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.013905 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6235185-3f02-4ff5-b304-1fa951c09248-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:38.013952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.013917 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6235185-3f02-4ff5-b304-1fa951c09248-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:38.013952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.013926 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr9sw\" (UniqueName: \"kubernetes.io/projected/e6235185-3f02-4ff5-b304-1fa951c09248-kube-api-access-sr9sw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:38.328545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.328458 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:38.735386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.735357 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" Nov 01 16:11:38.735822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.735387 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db" event={"ID":"e6235185-3f02-4ff5-b304-1fa951c09248","Type":"ContainerDied","Data":"6e9714d7b21651792680a9afe08be81cd5a2880886460e208ef24d9ee67f3b2e"} Nov 01 16:11:38.735822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.735451 4573 scope.go:122] "RemoveContainer" containerID="0ea0c27334103b1a780f41702e4bd2c6c71d6a8ec791f7cc89a97eff1efb652b" Nov 01 16:11:38.761908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.761881 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db"] Nov 01 16:11:38.767082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.767056 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-nl9db"] Nov 01 16:11:38.815978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.815932 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9"] Nov 01 16:11:38.817223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.817200 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" Nov 01 16:11:38.817302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.817226 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" Nov 01 16:11:38.817408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.817396 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" containerName="istio-proxy" Nov 01 16:11:38.825112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.825073 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.826953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.826929 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9"] Nov 01 16:11:38.857717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.857682 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.857930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.857796 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7698\" (UniqueName: \"kubernetes.io/projected/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-kube-api-access-c7698\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.857930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.857858 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.864063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.864039 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:11:38.864176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.864095 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:11:38.905427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.905400 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n"] Nov 01 16:11:38.910176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.910151 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:38.916407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.916384 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n"] Nov 01 16:11:38.960356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.960318 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcdjx\" (UniqueName: \"kubernetes.io/projected/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-kube-api-access-gcdjx\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:38.960508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.960488 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.960552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.960539 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7698\" (UniqueName: \"kubernetes.io/projected/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-kube-api-access-c7698\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.960605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.960594 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.960677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.960653 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:38.960762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.960736 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:38.961352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.961332 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.963033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.963010 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:38.968344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:38.968321 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7698\" (UniqueName: \"kubernetes.io/projected/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-kube-api-access-c7698\") pod \"maas-api-revoke-sub-e0dd02c1-rf9b9\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:39.062871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.062801 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcdjx\" (UniqueName: \"kubernetes.io/projected/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-kube-api-access-gcdjx\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:39.063022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.063005 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:39.063078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.063064 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:39.063641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.063620 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:39.065358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.065338 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:39.070469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.070446 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcdjx\" (UniqueName: \"kubernetes.io/projected/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-kube-api-access-gcdjx\") pod \"maas-api-revoke-keys-e2e-api-a-166778-xfs8n\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:39.149272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.149245 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:39.232191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.232157 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:39.437540 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:11:39.437502 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831 WatchSource:0}: Error finding container 7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831: Status 404 returned error can't find the container with id 7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831 Nov 01 16:11:39.438170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.438146 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9"] Nov 01 16:11:39.461227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.461204 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:39.582795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.581172 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n"] Nov 01 16:11:39.745320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.745231 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" event={"ID":"c13fbf8b-7440-4816-9ea4-49539c2b8bf0","Type":"ContainerStarted","Data":"7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831"} Nov 01 16:11:39.746576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.746552 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" event={"ID":"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4","Type":"ContainerStarted","Data":"f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10"} Nov 01 16:11:39.995511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.995454 4573 patch_prober.go:29] interesting pod/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:11:39.995511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:39.995497 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:11:40.281586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:40.281505 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:40.464855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:40.464818 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6235185-3f02-4ff5-b304-1fa951c09248" path="/var/lib/kubelet/pods/e6235185-3f02-4ff5-b304-1fa951c09248/volumes" Nov 01 16:11:40.758688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:40.758658 4573 generic.go:356] "Generic (PLEG): container finished" podID="e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4" containerID="2134c61c942a7782d1064b4655db0a30ce25350eb41668048ebda790ed58f8ab" exitCode=0 Nov 01 16:11:40.759112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:40.758759 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" event={"ID":"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4","Type":"ContainerDied","Data":"2134c61c942a7782d1064b4655db0a30ce25350eb41668048ebda790ed58f8ab"} Nov 01 16:11:40.760863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:40.760836 4573 generic.go:356] "Generic (PLEG): container finished" podID="c13fbf8b-7440-4816-9ea4-49539c2b8bf0" containerID="6eb3837d62cdcd7249b46d265cc4233003b7b1153950e69c6326d1982197f761" exitCode=0 Nov 01 16:11:40.760987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:40.760904 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" event={"ID":"c13fbf8b-7440-4816-9ea4-49539c2b8bf0","Type":"ContainerDied","Data":"6eb3837d62cdcd7249b46d265cc4233003b7b1153950e69c6326d1982197f761"} Nov 01 16:11:41.063786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:41.063735 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:41.756977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:41.756944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:42.062048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.062019 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:42.094989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.094959 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:42.174450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.174415 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-kube-api-access-c7698\" (UniqueName: \"kubernetes.io/projected/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-kube-api-access-c7698\") pod \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " Nov 01 16:11:42.174762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.174732 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-openshift-ca\") pod \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " Nov 01 16:11:42.174860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.174809 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-xks-ca\") pod \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\" (UID: \"c13fbf8b-7440-4816-9ea4-49539c2b8bf0\") " Nov 01 16:11:42.175092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.175069 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-openshift-ca" pod "c13fbf8b-7440-4816-9ea4-49539c2b8bf0" (UID: "c13fbf8b-7440-4816-9ea4-49539c2b8bf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:42.175479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.175462 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:42.176675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.176654 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-kube-api-access-c7698" pod "c13fbf8b-7440-4816-9ea4-49539c2b8bf0" (UID: "c13fbf8b-7440-4816-9ea4-49539c2b8bf0"). InnerVolumeSpecName "kube-api-access-c7698". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:42.176904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.176887 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-xks-ca" pod "c13fbf8b-7440-4816-9ea4-49539c2b8bf0" (UID: "c13fbf8b-7440-4816-9ea4-49539c2b8bf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:11:42.276446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.276357 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-openshift-ca\") pod \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " Nov 01 16:11:42.276807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.276682 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-kube-api-access-gcdjx\" (UniqueName: \"kubernetes.io/projected/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-kube-api-access-gcdjx\") pod \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " Nov 01 16:11:42.276807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.276736 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-xks-ca\") pod \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\" (UID: \"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4\") " Nov 01 16:11:42.277616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.277585 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:42.277725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.277622 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7698\" (UniqueName: \"kubernetes.io/projected/c13fbf8b-7440-4816-9ea4-49539c2b8bf0-kube-api-access-c7698\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:42.280588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.280558 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-openshift-ca" pod "e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4" (UID: "e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:42.280876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.280847 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-xks-ca" pod "e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4" (UID: "e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:11:42.282187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.282168 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-kube-api-access-gcdjx" pod "e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4" (UID: "e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4"). InnerVolumeSpecName "kube-api-access-gcdjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:42.295506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.295478 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:42.379094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.379055 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:42.379094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.379086 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcdjx\" (UniqueName: \"kubernetes.io/projected/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-kube-api-access-gcdjx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:42.379094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.379097 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:42.603849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.600843 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:42.714975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.714929 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:42.915693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.915004 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:42.972279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.972237 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" event={"ID":"c13fbf8b-7440-4816-9ea4-49539c2b8bf0","Type":"ContainerDied","Data":"7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831"} Nov 01 16:11:42.972566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.972548 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831" Nov 01 16:11:42.972771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.972739 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9" Nov 01 16:11:42.998114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.992228 4573 generic.go:356] "Generic (PLEG): container finished" podID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerID="493ecf574aa1c08e7d0cc8d8c8fd873f8af785e11999a2670f2c9199c5897530" exitCode=0 Nov 01 16:11:42.998114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.992356 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" event={"ID":"65cb6111-43a7-45b9-a7c6-d9398af6e215","Type":"ContainerDied","Data":"493ecf574aa1c08e7d0cc8d8c8fd873f8af785e11999a2670f2c9199c5897530"} Nov 01 16:11:42.998114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:42.992412 4573 scope.go:122] "RemoveContainer" containerID="6c83f536776ba4f42224e9b51fb97cdfe8ff7934db80e8b327d543fcb4590f0b" Nov 01 16:11:43.042003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.041963 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr"] Nov 01 16:11:43.048920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.048889 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-80c4c8-fmppr"] Nov 01 16:11:43.052957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.052923 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" event={"ID":"e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4","Type":"ContainerDied","Data":"f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10"} Nov 01 16:11:43.053126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.052981 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10" Nov 01 16:11:43.053126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.053060 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n" Nov 01 16:11:43.056576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.056545 4573 generic.go:356] "Generic (PLEG): container finished" podID="385cc464-bdff-46b2-a87a-70bc07951d15" containerID="c05dba7b8f340af41b3cfef122000000f8fbd6f50e20d679fec8c9d7e0432948" exitCode=0 Nov 01 16:11:43.056698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.056655 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" event={"ID":"385cc464-bdff-46b2-a87a-70bc07951d15","Type":"ContainerDied","Data":"c05dba7b8f340af41b3cfef122000000f8fbd6f50e20d679fec8c9d7e0432948"} Nov 01 16:11:43.060314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.060289 4573 generic.go:356] "Generic (PLEG): container finished" podID="1883148f-4863-492a-9da1-8787bc08c9c3" containerID="dabbc9eda559f6582cfdcf28be47e433c557ce80d517231ec47233a4ffc92283" exitCode=0 Nov 01 16:11:43.060534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.060513 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" event={"ID":"1883148f-4863-492a-9da1-8787bc08c9c3","Type":"ContainerDied","Data":"dabbc9eda559f6582cfdcf28be47e433c557ce80d517231ec47233a4ffc92283"} Nov 01 16:11:43.165365 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:43.163046 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:11:43.253092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.253055 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:43.300540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.300506 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:43.357663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.357635 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-data\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.357833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.357735 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1883148f-4863-492a-9da1-8787bc08c9c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1883148f-4863-492a-9da1-8787bc08c9c3-istio-podinfo\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.357833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.357804 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-istio-token\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-istio-token\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.357976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.357870 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-certs\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.357976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.357944 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1883148f-4863-492a-9da1-8787bc08c9c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1883148f-4863-492a-9da1-8787bc08c9c3-istiod-ca-cert\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.358270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.357981 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-data" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.358270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.358095 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-socket\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.358270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.358149 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-envoy\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.358435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.358313 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-kube-api-access-t7z7w\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-kube-api-access-t7z7w\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.358435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.358364 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-credential-socket\") pod \"1883148f-4863-492a-9da1-8787bc08c9c3\" (UID: \"1883148f-4863-492a-9da1-8787bc08c9c3\") " Nov 01 16:11:43.358866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.358563 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1883148f-4863-492a-9da1-8787bc08c9c3-istiod-ca-cert" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:43.359141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.359123 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.359222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.359145 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1883148f-4863-492a-9da1-8787bc08c9c3-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.359389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.359318 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-credential-socket" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.359454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.359439 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-socket" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.359696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.359665 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-certs" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.361821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.361783 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1883148f-4863-492a-9da1-8787bc08c9c3-istio-podinfo" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:11:43.366217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.366182 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-kube-api-access-t7z7w" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "kube-api-access-t7z7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.366408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.366381 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-istio-token" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.367486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.367466 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-envoy" pod "1883148f-4863-492a-9da1-8787bc08c9c3" (UID: "1883148f-4863-492a-9da1-8787bc08c9c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.371614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.371596 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:11:43.374382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.374364 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:43.459891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.459801 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-socket\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.460583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.459910 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-istio-token\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-istio-token\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.460583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.459954 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-envoy\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.460775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.460666 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-socket" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461044 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-credential-socket" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461512 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-credential-socket\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461629 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cb6111-43a7-45b9-a7c6-d9398af6e215-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cb6111-43a7-45b9-a7c6-d9398af6e215-istiod-ca-cert\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461670 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-podinfo\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461701 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-kube-api-access-zkzj8\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-kube-api-access-zkzj8\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461738 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-credential-socket\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461799 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-kube-api-access-xtb6r\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-kube-api-access-xtb6r\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461884 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/385cc464-bdff-46b2-a87a-70bc07951d15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cc464-bdff-46b2-a87a-70bc07951d15-istio-podinfo\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.461919 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/385cc464-bdff-46b2-a87a-70bc07951d15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cc464-bdff-46b2-a87a-70bc07951d15-istiod-ca-cert\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.462028 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-certs\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.462079 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-envoy\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.462184 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-certs\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.462339 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-socket\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.462400 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-certs" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.462438 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-token\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-token\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.462465 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-data\") pod \"65cb6111-43a7-45b9-a7c6-d9398af6e215\" (UID: \"65cb6111-43a7-45b9-a7c6-d9398af6e215\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.462508 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-data\") pod \"385cc464-bdff-46b2-a87a-70bc07951d15\" (UID: \"385cc464-bdff-46b2-a87a-70bc07951d15\") " Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.463117 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/385cc464-bdff-46b2-a87a-70bc07951d15-istiod-ca-cert" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.463471 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-certs" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.463665 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-socket" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.463911 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/385cc464-bdff-46b2-a87a-70bc07951d15-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464196 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7z7w\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-kube-api-access-t7z7w\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464212 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464227 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464241 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464255 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1883148f-4863-492a-9da1-8787bc08c9c3-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464269 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1883148f-4863-492a-9da1-8787bc08c9c3-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464283 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464297 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464312 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464327 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464341 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464355 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1883148f-4863-492a-9da1-8787bc08c9c3-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.464983 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cb6111-43a7-45b9-a7c6-d9398af6e215-istiod-ca-cert" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.465253 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-data" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.465833 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-kube-api-access-xtb6r" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "kube-api-access-xtb6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.466827 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-envoy" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.467189 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-data" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.469094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.467253 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-credential-socket" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.470921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.469393 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-kube-api-access-zkzj8" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "kube-api-access-zkzj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.470921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.470124 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-podinfo" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:11:43.470921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.470176 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-token" pod "65cb6111-43a7-45b9-a7c6-d9398af6e215" (UID: "65cb6111-43a7-45b9-a7c6-d9398af6e215"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.470921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.470260 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-istio-token" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:11:43.470921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.470260 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-envoy" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:11:43.471530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.471509 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/385cc464-bdff-46b2-a87a-70bc07951d15-istio-podinfo" pod "385cc464-bdff-46b2-a87a-70bc07951d15" (UID: "385cc464-bdff-46b2-a87a-70bc07951d15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:11:43.566948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.566909 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.566948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.566946 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.566965 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.566979 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.566992 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.567006 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/385cc464-bdff-46b2-a87a-70bc07951d15-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.567018 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65cb6111-43a7-45b9-a7c6-d9398af6e215-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.567032 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65cb6111-43a7-45b9-a7c6-d9398af6e215-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.567049 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkzj8\" (UniqueName: \"kubernetes.io/projected/65cb6111-43a7-45b9-a7c6-d9398af6e215-kube-api-access-zkzj8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.567082 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65cb6111-43a7-45b9-a7c6-d9398af6e215-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.567097 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtb6r\" (UniqueName: \"kubernetes.io/projected/385cc464-bdff-46b2-a87a-70bc07951d15-kube-api-access-xtb6r\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:43.567176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:43.567111 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/385cc464-bdff-46b2-a87a-70bc07951d15-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:11:44.073871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.073838 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn"] Nov 01 16:11:44.074496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.074469 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" Nov 01 16:11:44.074646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.074495 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m" event={"ID":"65cb6111-43a7-45b9-a7c6-d9398af6e215","Type":"ContainerDied","Data":"c5d25d2ea09f00ee199718bd2bfc9d0ee655fee2834fb09153f4221b6bad62f0"} Nov 01 16:11:44.074646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.074550 4573 scope.go:122] "RemoveContainer" containerID="493ecf574aa1c08e7d0cc8d8c8fd873f8af785e11999a2670f2c9199c5897530" Nov 01 16:11:44.076188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.076173 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" Nov 01 16:11:44.076292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.076201 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f" event={"ID":"385cc464-bdff-46b2-a87a-70bc07951d15","Type":"ContainerDied","Data":"d599f8af41da6ab57a28eac0bb7dd3dcc117f5ad0cf61fa4d18f937735d34af7"} Nov 01 16:11:44.077983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.077956 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" event={"ID":"1883148f-4863-492a-9da1-8787bc08c9c3","Type":"ContainerDied","Data":"c86ba456551ab4f7245ff31d2e4591ef8b8097ad15a6fa4c91d55d90f05b3463"} Nov 01 16:11:44.078097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.078005 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8" Nov 01 16:11:44.081543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.081519 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eaccb92-498jn"] Nov 01 16:11:44.089111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.089091 4573 scope.go:122] "RemoveContainer" containerID="c05dba7b8f340af41b3cfef122000000f8fbd6f50e20d679fec8c9d7e0432948" Nov 01 16:11:44.105309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.105290 4573 scope.go:122] "RemoveContainer" containerID="dabbc9eda559f6582cfdcf28be47e433c557ce80d517231ec47233a4ffc92283" Nov 01 16:11:44.142765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.142728 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f"] Nov 01 16:11:44.148796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.148766 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-p8m6f"] Nov 01 16:11:44.230529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.229528 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m"] Nov 01 16:11:44.241774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.236289 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:44.283855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.280030 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-kq55m"] Nov 01 16:11:44.288911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.288878 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8"] Nov 01 16:11:44.298794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.298735 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-76063d10-openshift-default-54bbfd48b9-4x8c8"] Nov 01 16:11:44.459912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.459878 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" path="/var/lib/kubelet/pods/1883148f-4863-492a-9da1-8787bc08c9c3/volumes" Nov 01 16:11:44.460422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.460406 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" path="/var/lib/kubelet/pods/385cc464-bdff-46b2-a87a-70bc07951d15/volumes" Nov 01 16:11:44.460937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.460920 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" path="/var/lib/kubelet/pods/65cb6111-43a7-45b9-a7c6-d9398af6e215/volumes" Nov 01 16:11:44.461649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.461633 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0061e80-728a-44c9-bb1f-1b09844b3f3a" path="/var/lib/kubelet/pods/b0061e80-728a-44c9-bb1f-1b09844b3f3a/volumes" Nov 01 16:11:44.462121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.462102 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098" path="/var/lib/kubelet/pods/f0c00c4c-4bdf-4f5f-9b40-a2c1ee8e4098/volumes" Nov 01 16:11:44.939719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:44.939683 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:45.663102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:45.663066 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:46.355489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:46.355455 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:46.777415 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:46.777372 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache]" Nov 01 16:11:46.982297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:46.982262 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:47.098837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:47.082038 4573 scope.go:122] "RemoveContainer" containerID="b876968002526433e91e84d15ed57a570ca9e45e75769cdd2823fa8f1bd961ae" Nov 01 16:11:47.142076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:47.142049 4573 scope.go:122] "RemoveContainer" containerID="98d025663e64a875f77297aab0705c34904035cd1937852b2e480eaf12f0b289" Nov 01 16:11:47.846937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:47.846901 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:48.560080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:48.560047 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:49.094221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:49.094185 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:49.194044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:49.194013 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:49.427267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:49.427231 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:49.524367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:49.524331 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:49.779326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:49.779229 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:49.879148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:49.879119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:50.001496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:50.001456 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:50.275720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:50.275655 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:50.378687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:50.378597 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:50.962950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:50.962902 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:51.306829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.300293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:51.431900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.430849 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 01 16:11:51.497782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.496896 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:51.537873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.537831 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:51.586996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.586886 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:51.650024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.649963 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:11:51.918137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.918104 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q"] Nov 01 16:11:51.932805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.919456 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c13fbf8b-7440-4816-9ea4-49539c2b8bf0" containerName="revoke-keys" Nov 01 16:11:51.933439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.933398 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13fbf8b-7440-4816-9ea4-49539c2b8bf0" containerName="revoke-keys" Nov 01 16:11:51.933628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.933617 4573 cpu_manager.go:414] "containerMap: removing container" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" Nov 01 16:11:51.933721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.933711 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" Nov 01 16:11:51.933848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.933837 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" Nov 01 16:11:51.933931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.933921 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" Nov 01 16:11:51.934037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.934025 4573 cpu_manager.go:414] "containerMap: removing container" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" Nov 01 16:11:51.934116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.934106 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" Nov 01 16:11:51.934215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.934205 4573 cpu_manager.go:414] "containerMap: removing container" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" Nov 01 16:11:51.934288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.934278 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" Nov 01 16:11:51.934386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.934376 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4" containerName="revoke-keys" Nov 01 16:11:51.940809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.940787 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4" containerName="revoke-keys" Nov 01 16:11:51.941254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.941233 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" Nov 01 16:11:51.942445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.941387 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cb6111-43a7-45b9-a7c6-d9398af6e215" containerName="istio-proxy" Nov 01 16:11:51.946450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.946431 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4" containerName="revoke-keys" Nov 01 16:11:51.946628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.946615 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13fbf8b-7440-4816-9ea4-49539c2b8bf0" containerName="revoke-keys" Nov 01 16:11:51.946740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.946730 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="385cc464-bdff-46b2-a87a-70bc07951d15" containerName="istio-proxy" Nov 01 16:11:51.946872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.946861 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1883148f-4863-492a-9da1-8787bc08c9c3" containerName="istio-proxy" Nov 01 16:11:51.949281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.942291 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:51.969788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.959849 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.021773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.970315 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2001b0-openshift-default-dockercfg-dwtwn\"" Nov 01 16:11:52.021773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:51.982482 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q"] Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215031 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-data\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215097 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-token\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215133 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-envoy\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215188 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-certs\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215296 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-podinfo\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215409 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215507 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-credential-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215568 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996fafaf-f9d4-41bf-a519-6c3f4575b473-istiod-ca-cert\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.253124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.215684 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfwzm\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-kube-api-access-nfwzm\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.328308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.328268 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996fafaf-f9d4-41bf-a519-6c3f4575b473-istiod-ca-cert\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.328977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.328951 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfwzm\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-kube-api-access-nfwzm\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.329233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.329210 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-data\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.329418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.329399 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-token\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.329547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.329530 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-envoy\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.329674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.329658 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-certs\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.329871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.329853 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-podinfo\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.330076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.330057 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.330254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.330234 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-credential-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.338795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.330570 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-data\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.338795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.329424 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996fafaf-f9d4-41bf-a519-6c3f4575b473-istiod-ca-cert\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.338795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.338278 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-credential-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.342140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.342115 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-certs\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.343616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.343571 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-socket\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.348081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.347244 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-envoy\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.348081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.347772 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-token\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.348824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.348802 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-podinfo\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.354812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.353571 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfwzm\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-kube-api-access-nfwzm\") pod \"e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.401077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.401044 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:52.517809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.517704 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:52.634490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.634455 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:52.960068 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:11:52.960033 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996fafaf_f9d4_41bf_a519_6c3f4575b473.slice/crio-bd673ee52229559e8691e905d21c385074f9ea30dbe71a856bf3f6efe58ff76f WatchSource:0}: Error finding container bd673ee52229559e8691e905d21c385074f9ea30dbe71a856bf3f6efe58ff76f: Status 404 returned error can't find the container with id bd673ee52229559e8691e905d21c385074f9ea30dbe71a856bf3f6efe58ff76f Nov 01 16:11:52.961103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.961079 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q"] Nov 01 16:11:52.963020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.962990 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:52.963121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.963051 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:52.963121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:52.963079 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:11:53.354949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:53.354859 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" event={"ID":"996fafaf-f9d4-41bf-a519-6c3f4575b473","Type":"ContainerStarted","Data":"092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2"} Nov 01 16:11:53.354949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:53.354926 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" event={"ID":"996fafaf-f9d4-41bf-a519-6c3f4575b473","Type":"ContainerStarted","Data":"bd673ee52229559e8691e905d21c385074f9ea30dbe71a856bf3f6efe58ff76f"} Nov 01 16:11:53.386418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:53.386365 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" podStartSLOduration=2.386350197 podStartE2EDuration="2.386350197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:53.378945732 +0000 UTC m=+1451.082111693" watchObservedRunningTime="2026-11-01 16:11:53.386350197 +0000 UTC m=+1451.089516132" Nov 01 16:11:53.637772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:53.635951 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:54.075612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:54.075537 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:54.201101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:54.197305 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:54.374771 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:54.374723 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:11:54.667848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:54.667614 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:54.690343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:54.690311 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:54.792357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:54.792331 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:55.373984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:55.373944 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:55.375115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:55.375093 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:11:55.783291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:55.783193 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:55.874213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:55.874181 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:56.039800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:56.039653 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:56.119719 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:56.119674 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:11:56.119970 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:56.119869 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache]" Nov 01 16:11:56.150834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:56.150804 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:56.795236 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:56.795186 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache]" Nov 01 16:11:57.008603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.008569 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:57.066408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.066328 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g"] Nov 01 16:11:57.071481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.071462 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:57.084093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.083431 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g"] Nov 01 16:11:57.084093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.083685 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2001b0\"" Nov 01 16:11:57.109974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.109918 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:57.110151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.110035 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2"] Nov 01 16:11:57.125080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.125045 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:57.134273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.134242 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2"] Nov 01 16:11:57.136153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.136119 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-2001b0-dockercfg-9t2mg\"" Nov 01 16:11:57.168642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.168611 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f2gw\" (UniqueName: \"kubernetes.io/projected/b364e1c7-df3f-4daa-b582-d11057bd4585-kube-api-access-2f2gw\") pod \"payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2\" (UID: \"b364e1c7-df3f-4daa-b582-d11057bd4585\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:57.168832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.168729 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndldr\" (UniqueName: \"kubernetes.io/projected/73f2b95e-884c-4b6c-8a93-4b47fccc47de-kube-api-access-ndldr\") pod \"payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g\" (UID: \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:57.168832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.168797 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364e1c7-df3f-4daa-b582-d11057bd4585-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2\" (UID: \"b364e1c7-df3f-4daa-b582-d11057bd4585\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:57.168939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.168912 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f2b95e-884c-4b6c-8a93-4b47fccc47de-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g\" (UID: \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:57.270651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.270612 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f2gw\" (UniqueName: \"kubernetes.io/projected/b364e1c7-df3f-4daa-b582-d11057bd4585-kube-api-access-2f2gw\") pod \"payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2\" (UID: \"b364e1c7-df3f-4daa-b582-d11057bd4585\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:57.271359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.271334 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndldr\" (UniqueName: \"kubernetes.io/projected/73f2b95e-884c-4b6c-8a93-4b47fccc47de-kube-api-access-ndldr\") pod \"payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g\" (UID: \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:57.275849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.275824 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364e1c7-df3f-4daa-b582-d11057bd4585-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2\" (UID: \"b364e1c7-df3f-4daa-b582-d11057bd4585\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:57.276097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.276077 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f2b95e-884c-4b6c-8a93-4b47fccc47de-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g\" (UID: \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:57.295499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.277081 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364e1c7-df3f-4daa-b582-d11057bd4585-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2\" (UID: \"b364e1c7-df3f-4daa-b582-d11057bd4585\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:57.297488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.297459 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f2b95e-884c-4b6c-8a93-4b47fccc47de-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g\" (UID: \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:57.320579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.320507 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f2gw\" (UniqueName: \"kubernetes.io/projected/b364e1c7-df3f-4daa-b582-d11057bd4585-kube-api-access-2f2gw\") pod \"payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2\" (UID: \"b364e1c7-df3f-4daa-b582-d11057bd4585\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:57.358809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.358770 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndldr\" (UniqueName: \"kubernetes.io/projected/73f2b95e-884c-4b6c-8a93-4b47fccc47de-kube-api-access-ndldr\") pod \"payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g\" (UID: \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:57.439113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.439079 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:57.454114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.452245 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:57.529914 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:11:57.528641 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:11:57.810048 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:11:57.810015 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f2b95e_884c_4b6c_8a93_4b47fccc47de.slice/crio-5fbe403d6fd94a012814852735e06bfed98227e285d29a1e2574935572faf0ba WatchSource:0}: Error finding container 5fbe403d6fd94a012814852735e06bfed98227e285d29a1e2574935572faf0ba: Status 404 returned error can't find the container with id 5fbe403d6fd94a012814852735e06bfed98227e285d29a1e2574935572faf0ba Nov 01 16:11:57.810789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.810762 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g"] Nov 01 16:11:57.832651 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:11:57.832623 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb364e1c7_df3f_4daa_b582_d11057bd4585.slice/crio-71a2aa5715aa470f4046b3656deaebef955a3f2a492c23c480fbe8920a8f9d84 WatchSource:0}: Error finding container 71a2aa5715aa470f4046b3656deaebef955a3f2a492c23c480fbe8920a8f9d84: Status 404 returned error can't find the container with id 71a2aa5715aa470f4046b3656deaebef955a3f2a492c23c480fbe8920a8f9d84 Nov 01 16:11:57.834686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:57.834666 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2"] Nov 01 16:11:58.378225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.378186 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:58.464561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.464526 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" event={"ID":"b364e1c7-df3f-4daa-b582-d11057bd4585","Type":"ContainerStarted","Data":"33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e"} Nov 01 16:11:58.464778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.464571 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" event={"ID":"b364e1c7-df3f-4daa-b582-d11057bd4585","Type":"ContainerStarted","Data":"71a2aa5715aa470f4046b3656deaebef955a3f2a492c23c480fbe8920a8f9d84"} Nov 01 16:11:58.464778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.464618 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:11:58.466262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.466238 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" event={"ID":"73f2b95e-884c-4b6c-8a93-4b47fccc47de","Type":"ContainerStarted","Data":"e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2"} Nov 01 16:11:58.466386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.466285 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" event={"ID":"73f2b95e-884c-4b6c-8a93-4b47fccc47de","Type":"ContainerStarted","Data":"5fbe403d6fd94a012814852735e06bfed98227e285d29a1e2574935572faf0ba"} Nov 01 16:11:58.466386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.466321 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:11:58.488920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.488880 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" podStartSLOduration=1.488868426 podStartE2EDuration="1.488868426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:58.483077056 +0000 UTC m=+1456.186243019" watchObservedRunningTime="2026-11-01 16:11:58.488868426 +0000 UTC m=+1456.192034362" Nov 01 16:11:58.505337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.505294 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" podStartSLOduration=1.5052830149999998 podStartE2EDuration="1.505283015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:11:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:11:58.498711828 +0000 UTC m=+1456.201877763" watchObservedRunningTime="2026-11-01 16:11:58.505283015 +0000 UTC m=+1456.208448950" Nov 01 16:11:58.958115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:58.958080 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:11:59.856272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:11:59.856241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:00.785549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:00.785512 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:01.638296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:01.638264 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:02.024700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:02.024617 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97"] Nov 01 16:12:02.034728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:02.034689 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b6afa92-ktt97"] Nov 01 16:12:02.462400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:02.462366 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fe8fbe-81ae-4977-8fe5-72ee75780644" path="/var/lib/kubelet/pods/82fe8fbe-81ae-4977-8fe5-72ee75780644/volumes" Nov 01 16:12:02.521638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:02.521602 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:02.648824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:02.648733 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:02.919865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:02.919826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:03.019207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:03.019175 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:03.665606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:03.665560 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:04.651642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:04.651553 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:05.365632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:05.365582 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:05.993606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:05.993573 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk"] Nov 01 16:12:05.998746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:05.998726 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.006852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.006824 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-375350-openshift-default-dockercfg-wzh7t\"" Nov 01 16:12:06.007326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.007303 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk"] Nov 01 16:12:06.129929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.129895 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.130134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.130104 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.130214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.130153 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.130359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.130332 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.130456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.130441 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.130518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.130478 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.130518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.130509 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.130616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.130594 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.130655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.130635 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bbw\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-kube-api-access-c9bbw\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.232677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.232640 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.232876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.232742 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.232876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.232839 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.232972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.232878 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.232972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.232924 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233072 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bbw\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-kube-api-access-c9bbw\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233179 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233183 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233203 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-credential-socket\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233248 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-certs\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233402 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233446 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233680 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-data\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.233874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.233692 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istiod-ca-cert\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.235540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.235517 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-podinfo\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.235637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.235595 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-envoy\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.243406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.243380 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bbw\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-kube-api-access-c9bbw\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.243592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.243567 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-token\") pod \"e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.259848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.259783 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:06.327104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.327072 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:06.723205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.723172 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk"] Nov 01 16:12:06.724044 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:12:06.724019 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa97b19f_b620_4e57_ab66_1abe674d0b0e.slice/crio-33017eab463c2ca1514b5caab4c58e9ef14406f51ec6c6cb931a0843acc23a35 WatchSource:0}: Error finding container 33017eab463c2ca1514b5caab4c58e9ef14406f51ec6c6cb931a0843acc23a35: Status 404 returned error can't find the container with id 33017eab463c2ca1514b5caab4c58e9ef14406f51ec6c6cb931a0843acc23a35 Nov 01 16:12:06.727157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.727122 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:06.727266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.727192 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:06.727266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:06.727220 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:06.892132 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:06.892090 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:12:07.151724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:07.151690 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:07.570316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:07.570238 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" event={"ID":"fa97b19f-b620-4e57-ab66-1abe674d0b0e","Type":"ContainerStarted","Data":"ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b"} Nov 01 16:12:07.570316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:07.570296 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" event={"ID":"fa97b19f-b620-4e57-ab66-1abe674d0b0e","Type":"ContainerStarted","Data":"33017eab463c2ca1514b5caab4c58e9ef14406f51ec6c6cb931a0843acc23a35"} Nov 01 16:12:07.627791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:07.627708 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" podStartSLOduration=2.627687403 podStartE2EDuration="2.627687403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:07.593245577 +0000 UTC m=+1465.296411513" watchObservedRunningTime="2026-11-01 16:12:07.627687403 +0000 UTC m=+1465.330853340" Nov 01 16:12:07.629640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:07.629614 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:07.717018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:07.716979 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:08.152396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:08.152353 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:08.331611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:08.328242 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:08.331842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:08.331662 4573 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 16:12:08.331842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:08.331718 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 16:12:08.897414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:08.897375 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:08.994399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:08.994352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:09.334823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:09.332213 4573 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 16:12:09.334823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:09.332277 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 16:12:09.498775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:09.498028 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:12:09.522774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:09.513944 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:12:09.754214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:09.754175 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:10.328081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:10.328047 4573 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 16:12:10.328287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:10.328124 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 16:12:10.635673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:10.635639 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:11.328422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:11.328388 4573 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 16:12:11.328585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:11.328459 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 16:12:11.584823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:11.584707 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:12.328148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:12.328101 4573 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 16:12:12.328681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:12.328186 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 16:12:12.393870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:12.393826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:12.716097 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:12.716054 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache]" Nov 01 16:12:13.172295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:13.172200 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:13.328026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:13.327999 4573 patch_prober.go:29] interesting pod/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Nov 01 16:12:13.328253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:13.328233 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Nov 01 16:12:13.729654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:13.729615 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:13.933780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:13.933420 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:14.046941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.046292 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:14.230102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.227626 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m"] Nov 01 16:12:14.240280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.240231 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-022f62-dpt9m"] Nov 01 16:12:14.306004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.305887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:14.333581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.333552 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:14.334627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.334606 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:14.335624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.335605 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:12:14.467637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.467589 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12db701c-3879-4426-a420-20cf4bc1632f" path="/var/lib/kubelet/pods/12db701c-3879-4426-a420-20cf4bc1632f/volumes" Nov 01 16:12:14.573697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:14.573607 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:15.245578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:15.245543 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:16.601774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:16.585785 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:16.769771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:16.750797 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:16.912734 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:16.912691 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:12:17.029684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:17.029639 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:17.720404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:17.720373 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:19.015963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:19.015930 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:19.140800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:19.140343 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:19.242357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:19.242325 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:20.065526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:20.065480 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:21.147805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.147769 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:21.298681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.298632 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:21.460966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.460889 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:21.861833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.861788 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:21.949576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.949538 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5"] Nov 01 16:12:21.957442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.957414 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:21.961346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.961318 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:12:21.961478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.961424 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:12:21.962436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.962413 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5"] Nov 01 16:12:21.965888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:21.965864 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:22.011377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.011305 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rntwt\" (UniqueName: \"kubernetes.io/projected/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-kube-api-access-rntwt\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.012031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.012002 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.012261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.012239 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.114571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.114477 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rntwt\" (UniqueName: \"kubernetes.io/projected/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-kube-api-access-rntwt\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.114571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.114533 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.114571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.114570 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.115551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.115512 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.118208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.118185 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.133359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.133334 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rntwt\" (UniqueName: \"kubernetes.io/projected/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-kube-api-access-rntwt\") pod \"maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.291918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.291884 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:22.392000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.391960 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:22.496998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.496962 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:22.627511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:22.627389 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5"] Nov 01 16:12:22.629804 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:12:22.629726 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfeeda8f_c8d1_4bc8_87d7_88a38fb4a141.slice/crio-8627a7972f48fa82d4fb6d16f7ae533f6566126fb492a83904184f6687593e3c WatchSource:0}: Error finding container 8627a7972f48fa82d4fb6d16f7ae533f6566126fb492a83904184f6687593e3c: Status 404 returned error can't find the container with id 8627a7972f48fa82d4fb6d16f7ae533f6566126fb492a83904184f6687593e3c Nov 01 16:12:23.027967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:23.027929 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" event={"ID":"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141","Type":"ContainerStarted","Data":"8627a7972f48fa82d4fb6d16f7ae533f6566126fb492a83904184f6687593e3c"} Nov 01 16:12:23.482341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:23.482305 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:24.038546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:24.038514 4573 generic.go:356] "Generic (PLEG): container finished" podID="dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141" containerID="31ad18d61240b3be32349091bde6253b66b8d50cfa8c4e3d04ba38cf4efba3d1" exitCode=0 Nov 01 16:12:24.038742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:24.038559 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" event={"ID":"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141","Type":"ContainerDied","Data":"31ad18d61240b3be32349091bde6253b66b8d50cfa8c4e3d04ba38cf4efba3d1"} Nov 01 16:12:24.272626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:24.272597 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:24.411859 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:24.411819 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:12:25.093776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.085535 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:25.294114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.294084 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:25.301019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.300989 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:25.424788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.414398 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-kube-api-access-rntwt\" (UniqueName: \"kubernetes.io/projected/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-kube-api-access-rntwt\") pod \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " Nov 01 16:12:25.425189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.425160 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-openshift-ca\") pod \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " Nov 01 16:12:25.436784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.425444 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-xks-ca\") pod \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\" (UID: \"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141\") " Nov 01 16:12:25.471998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.470265 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-openshift-ca" pod "dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141" (UID: "dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:25.538774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.536897 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-xks-ca" pod "dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141" (UID: "dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:25.550776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.546442 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:25.550776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.546477 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:25.610777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.592975 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-kube-api-access-rntwt" pod "dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141" (UID: "dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141"). InnerVolumeSpecName "kube-api-access-rntwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:25.652788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.652558 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rntwt\" (UniqueName: \"kubernetes.io/projected/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141-kube-api-access-rntwt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:25.858573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.858544 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s"] Nov 01 16:12:25.859350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.859334 4573 cpu_manager.go:414] "containerMap: removing container" podUID="dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141" containerName="revoke-keys" Nov 01 16:12:25.859425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.859351 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141" containerName="revoke-keys" Nov 01 16:12:25.859486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.859477 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141" containerName="revoke-keys" Nov 01 16:12:25.863831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.863807 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.871098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.871069 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f5dd93-openshift-default-dockercfg-h4q9h\"" Nov 01 16:12:25.876522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.876490 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s"] Nov 01 16:12:25.931067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.931030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:25.958243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958210 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-istio-token\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.958409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958303 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27rx7\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-kube-api-access-27rx7\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.958470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958447 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-envoy\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.958517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958486 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751315de-016b-4107-9b93-c3d4d139b9c0-istio-podinfo\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.958584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958572 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-certs\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.958634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958613 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-credential-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.958669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958656 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-data\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.958810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958787 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:25.958904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:25.958892 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751315de-016b-4107-9b93-c3d4d139b9c0-istiod-ca-cert\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.060836 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-certs\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.060914 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-credential-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.060985 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-data\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.061057 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.061152 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751315de-016b-4107-9b93-c3d4d139b9c0-istiod-ca-cert\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.061344 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-istio-token\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.061419 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27rx7\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-kube-api-access-27rx7\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.061475 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-envoy\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.062195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.061508 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751315de-016b-4107-9b93-c3d4d139b9c0-istio-podinfo\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.065549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.063235 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.065549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.063505 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-certs\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.065549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.063763 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-credential-socket\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.065549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.063995 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-data\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.065549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.064979 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751315de-016b-4107-9b93-c3d4d139b9c0-istiod-ca-cert\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.107226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.101171 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751315de-016b-4107-9b93-c3d4d139b9c0-istio-podinfo\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.121004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.120476 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27rx7\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-kube-api-access-27rx7\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.144774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.136369 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-envoy\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.155007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.154967 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" event={"ID":"dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141","Type":"ContainerDied","Data":"8627a7972f48fa82d4fb6d16f7ae533f6566126fb492a83904184f6687593e3c"} Nov 01 16:12:26.155201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.155033 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8627a7972f48fa82d4fb6d16f7ae533f6566126fb492a83904184f6687593e3c" Nov 01 16:12:26.155201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.155032 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-istio-token\") pod \"e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.155201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.155077 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5" Nov 01 16:12:26.186718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.186688 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:26.477954 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:12:26.477926 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751315de_016b_4107_9b93_c3d4d139b9c0.slice/crio-475e2e0d4c5b88144e2520fc119bc1804b87c194e7346ba4a936659b959659db WatchSource:0}: Error finding container 475e2e0d4c5b88144e2520fc119bc1804b87c194e7346ba4a936659b959659db: Status 404 returned error can't find the container with id 475e2e0d4c5b88144e2520fc119bc1804b87c194e7346ba4a936659b959659db Nov 01 16:12:26.479842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.479813 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s"] Nov 01 16:12:26.480819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.480787 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:26.480906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.480854 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:26.480906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.480883 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:12:26.943142 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:26.943090 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache]" Nov 01 16:12:26.950980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:26.950940 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:27.166650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:27.166617 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" event={"ID":"751315de-016b-4107-9b93-c3d4d139b9c0","Type":"ContainerStarted","Data":"815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d"} Nov 01 16:12:27.167091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:27.166673 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" event={"ID":"751315de-016b-4107-9b93-c3d4d139b9c0","Type":"ContainerStarted","Data":"475e2e0d4c5b88144e2520fc119bc1804b87c194e7346ba4a936659b959659db"} Nov 01 16:12:27.187227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:27.187191 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:27.188708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:27.188676 4573 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 16:12:27.188911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:27.188742 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 16:12:27.192227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:27.192189 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podStartSLOduration=2.1921769 podStartE2EDuration="2.1921769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:27.185902623 +0000 UTC m=+1484.889068554" watchObservedRunningTime="2026-11-01 16:12:27.1921769 +0000 UTC m=+1484.895342836" Nov 01 16:12:27.618048 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:27.618010 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:12:27.621780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:27.621740 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:28.187868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:28.187281 4573 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 16:12:28.187868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:28.187343 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 16:12:28.396066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:28.396030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:29.180775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:29.172228 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:29.187215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:29.187182 4573 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 16:12:29.187389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:29.187245 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 16:12:29.894971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:29.894937 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv"] Nov 01 16:12:29.895569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:29.895137 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" containerID="cri-o://aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049" gracePeriod=30 Nov 01 16:12:29.902918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:29.902893 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:30.187289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:30.187193 4573 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 16:12:30.187564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:30.187530 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 16:12:30.657714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:30.657675 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:30.725627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:30.725595 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:30.826115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:30.826063 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:31.187283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:31.187244 4573 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 16:12:31.187931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:31.187895 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 16:12:31.308108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:31.308070 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:31.611955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:31.611897 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:32.212251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:32.212178 4573 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 16:12:32.213506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:32.212794 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 16:12:32.234363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:32.234339 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:32.316958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:32.316927 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:32.446529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:32.446488 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:33.187108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:33.187075 4573 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" start-of-body= Nov 01 16:12:33.187291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:33.187129 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.181:15021/healthz/ready\": dial tcp 10.133.0.181:15021: connect: connection refused" Nov 01 16:12:33.252931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:33.252879 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:33.988834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:33.988800 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:34.192895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:34.192862 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:34.193334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:34.193313 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:34.193953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:34.193933 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:12:34.760689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:34.760651 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:34.858483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:34.858450 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:35.388999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.388962 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:12:35.431124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.431093 4573 generic.go:356] "Generic (PLEG): container finished" podID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerID="aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049" exitCode=0 Nov 01 16:12:35.431310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.431217 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" Nov 01 16:12:35.431310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.431268 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" event={"ID":"5dcbf4a1-4324-446c-911a-b0836fb21ba5","Type":"ContainerDied","Data":"aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049"} Nov 01 16:12:35.431445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.431343 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv" event={"ID":"5dcbf4a1-4324-446c-911a-b0836fb21ba5","Type":"ContainerDied","Data":"c3e1db775020390b825f4c5378fdffe99af6e6ef507f921a757da5427a0eda0b"} Nov 01 16:12:35.431445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.431376 4573 scope.go:122] "RemoveContainer" containerID="aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049" Nov 01 16:12:35.452506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.452477 4573 scope.go:122] "RemoveContainer" containerID="aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049" Nov 01 16:12:35.462151 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:35.462118 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049\": container with ID starting with aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049 not found: ID does not exist" containerID="aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049" Nov 01 16:12:35.462278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.462158 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049"} err="failed to get container status \"aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049\": rpc error: code = NotFound desc = could not find container \"aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049\": container with ID starting with aab4b4380f564bffda62f23861ebdc989f3886e7b9f1a07e51d8e63ea34f2049 not found: ID does not exist" Nov 01 16:12:35.471136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.471112 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:35.531683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.531653 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-token\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-token\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.531998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.531970 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-certs\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.532118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532081 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istiod-ca-cert\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.532182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532141 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-credential-socket\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.532259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532236 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-certs" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:35.532375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532291 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-data\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.532513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532494 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-socket\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.532568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532517 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-credential-socket" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:35.532568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532532 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-podinfo\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.532568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532545 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istiod-ca-cert" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:35.532745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532727 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-data" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:35.532839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532811 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-envoy\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.532927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.532909 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-kube-api-access-nkhr4\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-kube-api-access-nkhr4\") pod \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\" (UID: \"5dcbf4a1-4324-446c-911a-b0836fb21ba5\") " Nov 01 16:12:35.533308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.533280 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-socket" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:35.534668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.533734 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.534893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.534849 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.534893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.534869 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.534893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.534879 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.536895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.536872 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-envoy" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:35.537000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.536982 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-token" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:35.537269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.537247 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-kube-api-access-nkhr4" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "kube-api-access-nkhr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:35.537486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.537466 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-podinfo" pod "5dcbf4a1-4324-446c-911a-b0836fb21ba5" (UID: "5dcbf4a1-4324-446c-911a-b0836fb21ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:12:35.643876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.643834 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.643876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.643876 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.643876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.643891 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.644208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.643905 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkhr4\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-kube-api-access-nkhr4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.644208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.643921 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dcbf4a1-4324-446c-911a-b0836fb21ba5-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:35.671779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.668949 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:35.863377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.862804 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv"] Nov 01 16:12:35.902841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.900413 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-kh5rv"] Nov 01 16:12:35.921772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.921622 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n"] Nov 01 16:12:35.994470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:35.988736 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-166778-xfs8n"] Nov 01 16:12:36.465658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:36.465617 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" path="/var/lib/kubelet/pods/5dcbf4a1-4324-446c-911a-b0836fb21ba5/volumes" Nov 01 16:12:36.466368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:36.466348 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4" path="/var/lib/kubelet/pods/e0b12ac6-03f2-47d4-9cdf-f9de0a9244e4/volumes" Nov 01 16:12:37.037647 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:37.037604 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:12:37.204711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:37.204682 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:37.384777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:37.367811 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:38.107308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:38.107277 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:38.899877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:38.899845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:39.564023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:39.563985 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:40.524982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:40.522584 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:41.314327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:41.314296 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:42.259784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:42.248968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:42.363820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:42.363517 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:42.449357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:42.449305 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:42.620409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:42.620367 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:42.719394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:42.681528 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh"] Nov 01 16:12:42.725787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:42.720030 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" containerID="cri-o://3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024" gracePeriod=30 Nov 01 16:12:43.021938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.021852 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm"] Nov 01 16:12:43.023286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.023258 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" Nov 01 16:12:43.023447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.023434 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" Nov 01 16:12:43.023736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.023722 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dcbf4a1-4324-446c-911a-b0836fb21ba5" containerName="istio-proxy" Nov 01 16:12:43.039883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.039736 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.041415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.041385 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm"] Nov 01 16:12:43.048655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.048628 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:12:43.049058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.049040 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:12:43.074614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.074576 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.074958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.074934 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.075391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.075370 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x6px\" (UniqueName: \"kubernetes.io/projected/90921542-85ee-45b1-ab3f-f257e1ff5c06-kube-api-access-9x6px\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.087183 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:43.087140 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13fbf8b_7440_4816_9ea4_49539c2b8bf0.slice/crio-7df37be9d26414228eaf9de7d28d0ec58126875da0115d389e203b6a92810831\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b12ac6_03f2_47d4_9cdf_f9de0a9244e4.slice/crio-f2b0e26c5daf577d9277980cf55e94ab1a00596574948b2f33cf5eec88805b10\": RecentStats: unable to find data in memory cache]" Nov 01 16:12:43.115810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.115741 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:12:43.120711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.120684 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:12:43.128341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.128319 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:12:43.133243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.133221 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:12:43.178096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.178057 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x6px\" (UniqueName: \"kubernetes.io/projected/90921542-85ee-45b1-ab3f-f257e1ff5c06-kube-api-access-9x6px\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.178286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.178188 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.178286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.178251 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.178945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.178925 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.180832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.180808 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.186211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.186180 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x6px\" (UniqueName: \"kubernetes.io/projected/90921542-85ee-45b1-ab3f-f257e1ff5c06-kube-api-access-9x6px\") pod \"maas-api-revoke-keys-e2e-api-b-26310f-lt6dm\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.376919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.376887 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:43.440585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.440555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:43.660424 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:12:43.660391 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90921542_85ee_45b1_ab3f_f257e1ff5c06.slice/crio-962974c8084e3d9caf0f857d682690db79b4fbdd8a9b30aaddbbc7c023ab4e34 WatchSource:0}: Error finding container 962974c8084e3d9caf0f857d682690db79b4fbdd8a9b30aaddbbc7c023ab4e34: Status 404 returned error can't find the container with id 962974c8084e3d9caf0f857d682690db79b4fbdd8a9b30aaddbbc7c023ab4e34 Nov 01 16:12:43.660886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.660863 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm"] Nov 01 16:12:43.734208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:43.734174 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" event={"ID":"90921542-85ee-45b1-ab3f-f257e1ff5c06","Type":"ContainerStarted","Data":"962974c8084e3d9caf0f857d682690db79b4fbdd8a9b30aaddbbc7c023ab4e34"} Nov 01 16:12:44.304708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:44.304648 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:44.744373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:44.744333 4573 generic.go:356] "Generic (PLEG): container finished" podID="90921542-85ee-45b1-ab3f-f257e1ff5c06" containerID="fba99e310295e36aa3b3647166538e5953dde89f8387788db1141b1b8d6620e2" exitCode=0 Nov 01 16:12:44.744742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:44.744419 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" event={"ID":"90921542-85ee-45b1-ab3f-f257e1ff5c06","Type":"ContainerDied","Data":"fba99e310295e36aa3b3647166538e5953dde89f8387788db1141b1b8d6620e2"} Nov 01 16:12:44.991261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:44.991152 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:45.553210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:45.553179 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:45.728965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:45.728922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:46.010260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.010237 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:46.011325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.011300 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:46.174308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.174227 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-openshift-ca\") pod \"90921542-85ee-45b1-ab3f-f257e1ff5c06\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " Nov 01 16:12:46.174308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.174282 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-xks-ca\") pod \"90921542-85ee-45b1-ab3f-f257e1ff5c06\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " Nov 01 16:12:46.174501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.174448 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90921542-85ee-45b1-ab3f-f257e1ff5c06-kube-api-access-9x6px\" (UniqueName: \"kubernetes.io/projected/90921542-85ee-45b1-ab3f-f257e1ff5c06-kube-api-access-9x6px\") pod \"90921542-85ee-45b1-ab3f-f257e1ff5c06\" (UID: \"90921542-85ee-45b1-ab3f-f257e1ff5c06\") " Nov 01 16:12:46.174624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.174590 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-openshift-ca" pod "90921542-85ee-45b1-ab3f-f257e1ff5c06" (UID: "90921542-85ee-45b1-ab3f-f257e1ff5c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:46.175285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.175265 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:46.176486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.176463 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-xks-ca" pod "90921542-85ee-45b1-ab3f-f257e1ff5c06" (UID: "90921542-85ee-45b1-ab3f-f257e1ff5c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:12:46.176623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.176578 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90921542-85ee-45b1-ab3f-f257e1ff5c06-kube-api-access-9x6px" pod "90921542-85ee-45b1-ab3f-f257e1ff5c06" (UID: "90921542-85ee-45b1-ab3f-f257e1ff5c06"). InnerVolumeSpecName "kube-api-access-9x6px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:46.276774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.276732 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x6px\" (UniqueName: \"kubernetes.io/projected/90921542-85ee-45b1-ab3f-f257e1ff5c06-kube-api-access-9x6px\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:46.276774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.276771 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90921542-85ee-45b1-ab3f-f257e1ff5c06-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:46.462315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.462203 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:46.618776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.617310 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:46.715415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.715312 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:46.835037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.833933 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" event={"ID":"90921542-85ee-45b1-ab3f-f257e1ff5c06","Type":"ContainerDied","Data":"962974c8084e3d9caf0f857d682690db79b4fbdd8a9b30aaddbbc7c023ab4e34"} Nov 01 16:12:46.835037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.834004 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="962974c8084e3d9caf0f857d682690db79b4fbdd8a9b30aaddbbc7c023ab4e34" Nov 01 16:12:46.835037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:46.834072 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm" Nov 01 16:12:47.041019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:47.040939 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:47.203960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:47.203921 4573 scope.go:122] "RemoveContainer" containerID="2ce533b2958f03378e68fbfede326c0fa6f8860e755dfefa9356dfa28d30ce63" Nov 01 16:12:47.215139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:47.215118 4573 scope.go:122] "RemoveContainer" containerID="a481b9411844996d7e4db5bce5d12d35fef7ab7c97d9265be9095779610679df" Nov 01 16:12:47.295594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:47.295509 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:47.444185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:47.444156 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:47.546179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:47.546101 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:48.221482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.221455 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:12:48.275976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.275902 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6590e665-4fc6-42db-a481-775a1fd4d403-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6590e665-4fc6-42db-a481-775a1fd4d403-istio-podinfo\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.275995 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-envoy\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276172 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6590e665-4fc6-42db-a481-775a1fd4d403-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6590e665-4fc6-42db-a481-775a1fd4d403-istiod-ca-cert\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276220 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-data\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276296 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-istio-token\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-istio-token\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276390 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-certs\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276461 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-socket\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276486 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6590e665-4fc6-42db-a481-775a1fd4d403-istiod-ca-cert" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:12:48.276625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276501 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-kube-api-access-mv8lc\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-kube-api-access-mv8lc\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276532 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-data" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:48.276625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276565 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-credential-socket\") pod \"6590e665-4fc6-42db-a481-775a1fd4d403\" (UID: \"6590e665-4fc6-42db-a481-775a1fd4d403\") " Nov 01 16:12:48.276821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276633 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-certs" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:48.276821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276735 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-socket" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:48.276941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.276908 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-credential-socket" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:48.277290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.277265 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6590e665-4fc6-42db-a481-775a1fd4d403-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.277290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.277290 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.277468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.277299 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.277468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.277308 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.277468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.277317 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.278645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.278620 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6590e665-4fc6-42db-a481-775a1fd4d403-istio-podinfo" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:12:48.278775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.278690 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-envoy" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:12:48.278841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.278811 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-istio-token" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:48.279019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.278992 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-kube-api-access-mv8lc" pod "6590e665-4fc6-42db-a481-775a1fd4d403" (UID: "6590e665-4fc6-42db-a481-775a1fd4d403"). InnerVolumeSpecName "kube-api-access-mv8lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:12:48.348763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.348716 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:48.379154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.379126 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6590e665-4fc6-42db-a481-775a1fd4d403-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.379154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.379152 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6590e665-4fc6-42db-a481-775a1fd4d403-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.379340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.379164 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.379340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.379174 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv8lc\" (UniqueName: \"kubernetes.io/projected/6590e665-4fc6-42db-a481-775a1fd4d403-kube-api-access-mv8lc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:12:48.870507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.870477 4573 generic.go:356] "Generic (PLEG): container finished" podID="6590e665-4fc6-42db-a481-775a1fd4d403" containerID="3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024" exitCode=0 Nov 01 16:12:48.870717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.870524 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" Nov 01 16:12:48.870717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.870561 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" event={"ID":"6590e665-4fc6-42db-a481-775a1fd4d403","Type":"ContainerDied","Data":"3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024"} Nov 01 16:12:48.870717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.870624 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh" event={"ID":"6590e665-4fc6-42db-a481-775a1fd4d403","Type":"ContainerDied","Data":"8f0ee02128968df3c38a7b1dd6ef99a54ed2c234c169d0de73ef849491bac5a5"} Nov 01 16:12:48.870717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.870657 4573 scope.go:122] "RemoveContainer" containerID="3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024" Nov 01 16:12:48.882107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.882087 4573 scope.go:122] "RemoveContainer" containerID="56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81" Nov 01 16:12:48.893925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.893892 4573 scope.go:122] "RemoveContainer" containerID="3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024" Nov 01 16:12:48.894286 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:48.894264 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024\": container with ID starting with 3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024 not found: ID does not exist" containerID="3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024" Nov 01 16:12:48.894404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.894297 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024"} err="failed to get container status \"3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024\": rpc error: code = NotFound desc = could not find container \"3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024\": container with ID starting with 3c09fdc58d4d5b2f1acedfffa09e6d1dff8e045990b04bb7d37d91582e9de024 not found: ID does not exist" Nov 01 16:12:48.894404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.894323 4573 scope.go:122] "RemoveContainer" containerID="56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81" Nov 01 16:12:48.894659 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:48.894637 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81\": container with ID starting with 56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81 not found: ID does not exist" containerID="56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81" Nov 01 16:12:48.894697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.894665 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81"} err="failed to get container status \"56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81\": rpc error: code = NotFound desc = could not find container \"56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81\": container with ID starting with 56454dcaf9296bd7b53ad354359ec43bfc0e08b067e29f4341529f0d2c206a81 not found: ID does not exist" Nov 01 16:12:48.897126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.897108 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh"] Nov 01 16:12:48.901489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:48.901468 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-166778-openshift-default-869f777c67-4ddgh"] Nov 01 16:12:49.151640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:49.151565 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:50.062781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:50.062713 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:50.382869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:50.382841 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:12:50.460409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:50.460376 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" path="/var/lib/kubelet/pods/6590e665-4fc6-42db-a481-775a1fd4d403/volumes" Nov 01 16:12:50.949532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:50.949499 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:51.070799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:51.070737 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:51.186064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:51.186033 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:51.868575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:51.868532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:52.124736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:52.124618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:52.602823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:52.594215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:52.949926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:52.949890 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:53.056117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:53.056076 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:54.347965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:54.347888 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:54.613672 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:12:54.613629 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:12:55.043590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.043500 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:55.133037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.133007 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq"] Nov 01 16:12:55.135932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.134226 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:55.136471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.136446 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" Nov 01 16:12:55.136604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.136591 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" Nov 01 16:12:55.136697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.136688 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" Nov 01 16:12:55.136802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.136791 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" Nov 01 16:12:55.136987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.136974 4573 cpu_manager.go:414] "containerMap: removing container" podUID="90921542-85ee-45b1-ab3f-f257e1ff5c06" containerName="revoke-keys" Nov 01 16:12:55.137067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.137058 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90921542-85ee-45b1-ab3f-f257e1ff5c06" containerName="revoke-keys" Nov 01 16:12:55.137337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.137323 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" Nov 01 16:12:55.137439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.137428 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6590e665-4fc6-42db-a481-775a1fd4d403" containerName="istio-proxy" Nov 01 16:12:55.137540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.137530 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="90921542-85ee-45b1-ab3f-f257e1ff5c06" containerName="revoke-keys" Nov 01 16:12:55.143972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.143952 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq"] Nov 01 16:12:55.144693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.144083 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:55.153940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.153920 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-99e837-dockercfg-6fd9s\"" Nov 01 16:12:55.154062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.154009 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-99e837\"" Nov 01 16:12:55.274327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.256615 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9"] Nov 01 16:12:55.274327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.259279 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73f29d0-5265-499b-af4c-a47dec1c99dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq\" (UID: \"c73f29d0-5265-499b-af4c-a47dec1c99dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:55.274327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.259377 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67n2t\" (UniqueName: \"kubernetes.io/projected/c73f29d0-5265-499b-af4c-a47dec1c99dd-kube-api-access-67n2t\") pod \"payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq\" (UID: \"c73f29d0-5265-499b-af4c-a47dec1c99dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:55.304856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.288047 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:55.384070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.381236 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73f29d0-5265-499b-af4c-a47dec1c99dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq\" (UID: \"c73f29d0-5265-499b-af4c-a47dec1c99dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:55.393904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.393863 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67n2t\" (UniqueName: \"kubernetes.io/projected/c73f29d0-5265-499b-af4c-a47dec1c99dd-kube-api-access-67n2t\") pod \"payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq\" (UID: \"c73f29d0-5265-499b-af4c-a47dec1c99dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:55.405454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.397315 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73f29d0-5265-499b-af4c-a47dec1c99dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq\" (UID: \"c73f29d0-5265-499b-af4c-a47dec1c99dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:55.461296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.457283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:55.526145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.526100 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfvt\" (UniqueName: \"kubernetes.io/projected/da65f65f-be21-4a00-bb5c-f0c3f183aee1-kube-api-access-bhfvt\") pod \"payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9\" (UID: \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:55.526874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.526849 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da65f65f-be21-4a00-bb5c-f0c3f183aee1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9\" (UID: \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:55.612830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.593991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:55.682352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.654180 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfvt\" (UniqueName: \"kubernetes.io/projected/da65f65f-be21-4a00-bb5c-f0c3f183aee1-kube-api-access-bhfvt\") pod \"payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9\" (UID: \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:55.696100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.696067 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da65f65f-be21-4a00-bb5c-f0c3f183aee1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9\" (UID: \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:55.712880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.690340 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfvt\" (UniqueName: \"kubernetes.io/projected/da65f65f-be21-4a00-bb5c-f0c3f183aee1-kube-api-access-bhfvt\") pod \"payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9\" (UID: \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:55.740776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.714246 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da65f65f-be21-4a00-bb5c-f0c3f183aee1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9\" (UID: \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:55.831791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.806689 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67n2t\" (UniqueName: \"kubernetes.io/projected/c73f29d0-5265-499b-af4c-a47dec1c99dd-kube-api-access-67n2t\") pod \"payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq\" (UID: \"c73f29d0-5265-499b-af4c-a47dec1c99dd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:55.848791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.844835 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9"] Nov 01 16:12:55.962833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:55.921083 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:56.116821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:56.114661 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:56.247883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:56.246445 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:12:56.498944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:56.491303 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:57.015820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.015705 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9"] Nov 01 16:12:57.022040 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:12:57.022005 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda65f65f_be21_4a00_bb5c_f0c3f183aee1.slice/crio-897f588950ad57f94221e7decfee521497fe26afbd42db8b6261a44e4e641b4a WatchSource:0}: Error finding container 897f588950ad57f94221e7decfee521497fe26afbd42db8b6261a44e4e641b4a: Status 404 returned error can't find the container with id 897f588950ad57f94221e7decfee521497fe26afbd42db8b6261a44e4e641b4a Nov 01 16:12:57.038312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.038285 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq"] Nov 01 16:12:57.271542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.271460 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:57.528335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.528244 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" event={"ID":"da65f65f-be21-4a00-bb5c-f0c3f183aee1","Type":"ContainerStarted","Data":"d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e"} Nov 01 16:12:57.528894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.528873 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:12:57.529017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.528994 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" event={"ID":"da65f65f-be21-4a00-bb5c-f0c3f183aee1","Type":"ContainerStarted","Data":"897f588950ad57f94221e7decfee521497fe26afbd42db8b6261a44e4e641b4a"} Nov 01 16:12:57.531489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.531458 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" event={"ID":"c73f29d0-5265-499b-af4c-a47dec1c99dd","Type":"ContainerStarted","Data":"2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a"} Nov 01 16:12:57.531606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.531510 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" event={"ID":"c73f29d0-5265-499b-af4c-a47dec1c99dd","Type":"ContainerStarted","Data":"45c5d7a04528696a6f0eb08987056ddc6d8071f301a2b0a7e1c0fa31f9cd383c"} Nov 01 16:12:57.531676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.531666 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:12:57.549673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.549631 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" podStartSLOduration=2.54961901 podStartE2EDuration="2.54961901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:57.544424363 +0000 UTC m=+1515.247590303" watchObservedRunningTime="2026-11-01 16:12:57.54961901 +0000 UTC m=+1515.252784950" Nov 01 16:12:57.592776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:57.587194 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" podStartSLOduration=2.587174576 podStartE2EDuration="2.587174576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:12:57.560070144 +0000 UTC m=+1515.263236086" watchObservedRunningTime="2026-11-01 16:12:57.587174576 +0000 UTC m=+1515.290340515" Nov 01 16:12:58.012543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:58.011587 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:58.613780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:58.612232 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:58.832763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:58.832691 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:59.064835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.064655 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:59.436641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.433704 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z"] Nov 01 16:12:59.442427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.441335 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.449347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.449317 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-10b00a-openshift-default-dockercfg-tv6qt\"" Nov 01 16:12:59.460327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.460297 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z"] Nov 01 16:12:59.496681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.496630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:12:59.676771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.630769 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-envoy\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.759459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.713872 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-token\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.759459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.713985 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-credential-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.759459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.714068 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-podinfo\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.759459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.714151 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-data\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.759459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.714187 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-certs\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.759459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.714273 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sxzx\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-kube-api-access-8sxzx\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.759459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.714367 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.759459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.714635 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istiod-ca-cert\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.837910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.837858 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istiod-ca-cert\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.838216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.838196 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-envoy\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.838456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.838402 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-token\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.838692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.838669 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-credential-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.855745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.855716 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-podinfo\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.904463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.856036 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-data\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.968031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.965781 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-certs\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.968031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.965898 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sxzx\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-kube-api-access-8sxzx\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.968031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.966001 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.968031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.966633 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.968031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.871129 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-data\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.968031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.966956 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-certs\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.968031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.855598 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-credential-socket\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.968031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.854270 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istiod-ca-cert\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:12:59.985682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:12:59.985606 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:00.004952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.004883 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-envoy\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:00.016029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.015553 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-token\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:00.074784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.065612 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sxzx\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-kube-api-access-8sxzx\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:00.122781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.088348 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-podinfo\") pod \"e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:00.122781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.088981 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:00.224138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.224100 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:00.543874 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:13:00.543840 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c921e79_5128_4b20_bdc3_302e60b3e5d6.slice/crio-dfac2842dd817e8e59b807e2b139ac04c35277d3c799079d16f24bb36a831b4f WatchSource:0}: Error finding container dfac2842dd817e8e59b807e2b139ac04c35277d3c799079d16f24bb36a831b4f: Status 404 returned error can't find the container with id dfac2842dd817e8e59b807e2b139ac04c35277d3c799079d16f24bb36a831b4f Nov 01 16:13:00.544013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.543947 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z"] Nov 01 16:13:00.547060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.547023 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:00.547189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.547100 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:00.547189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.547155 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:00.673985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.673919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:00.945393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.945349 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" event={"ID":"9c921e79-5128-4b20-bdc3-302e60b3e5d6","Type":"ContainerStarted","Data":"6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d"} Nov 01 16:13:00.945945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.945414 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" event={"ID":"9c921e79-5128-4b20-bdc3-302e60b3e5d6","Type":"ContainerStarted","Data":"dfac2842dd817e8e59b807e2b139ac04c35277d3c799079d16f24bb36a831b4f"} Nov 01 16:13:00.978813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:00.964469 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podStartSLOduration=1.9644562570000002 podStartE2EDuration="1.964456257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:12:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:00.963631831 +0000 UTC m=+1518.666797769" watchObservedRunningTime="2026-11-01 16:13:00.964456257 +0000 UTC m=+1518.667622193" Nov 01 16:13:01.224569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:01.224478 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:01.226883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:01.226669 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:13:01.226883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:01.226730 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 16:13:01.325924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:01.325873 4573 patch_prober.go:29] interesting pod/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:13:01.326130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:01.325935 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:01.474174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:01.474133 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:02.231370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:02.231331 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:02.465821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:02.465739 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:02.968843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:02.968805 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:02.969993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:02.969968 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:13:03.280173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:03.280090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:03.376882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:03.376847 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:03.539246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:03.539132 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:04.259357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:04.259320 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:04.577513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:04.577229 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:04.667502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:04.667467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:05.264125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:05.264093 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:06.042155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.042118 4573 generic.go:356] "Generic (PLEG): container finished" podID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerID="61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33" exitCode=0 Nov 01 16:13:06.042626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.042193 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" event={"ID":"3900c769-1183-45c0-aa46-0cb33ddc5778","Type":"ContainerDied","Data":"61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33"} Nov 01 16:13:06.042829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.042811 4573 scope.go:122] "RemoveContainer" containerID="61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33" Nov 01 16:13:06.045395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.045363 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:06.045498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.045436 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:06.045498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.045466 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:06.046165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.046145 4573 generic.go:356] "Generic (PLEG): container finished" podID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerID="092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2" exitCode=0 Nov 01 16:13:06.046471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.046170 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" event={"ID":"996fafaf-f9d4-41bf-a519-6c3f4575b473","Type":"ContainerDied","Data":"092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2"} Nov 01 16:13:06.046918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.046899 4573 scope.go:122] "RemoveContainer" containerID="092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2" Nov 01 16:13:06.049227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.049197 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:06.049336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.049285 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:06.049472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.049345 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:13:06.146655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.145387 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc"] Nov 01 16:13:06.146655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.145601 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" containerID="cri-o://56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60" gracePeriod=30 Nov 01 16:13:06.158308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.157251 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n"] Nov 01 16:13:06.158308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.157478 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" containerID="cri-o://19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4" gracePeriod=30 Nov 01 16:13:06.214564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:06.214534 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:07.018783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.013399 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:07.079764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.079712 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" event={"ID":"3900c769-1183-45c0-aa46-0cb33ddc5778","Type":"ContainerStarted","Data":"4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae"} Nov 01 16:13:07.083847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.083814 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" event={"ID":"996fafaf-f9d4-41bf-a519-6c3f4575b473","Type":"ContainerStarted","Data":"2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9"} Nov 01 16:13:07.635349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.635320 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:13:07.636141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.636113 4573 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:13:07.636265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.636160 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 16:13:07.810790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.808830 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:07.992841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.992129 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:13:07.992841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.992247 4573 patch_prober.go:29] interesting pod/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:13:07.992841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:07.992288 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 16:13:08.547433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:08.547405 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:13:08.643839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:08.641721 4573 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:13:08.643839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:08.642650 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 16:13:08.709780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:08.704671 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:08.988776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:08.988184 4573 patch_prober.go:29] interesting pod/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:13:08.988776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:08.988258 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 16:13:09.495774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:09.493744 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:09.637845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:09.636115 4573 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:13:09.637845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:09.636192 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 16:13:09.944520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:09.944485 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:13:10.000785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:09.993169 4573 patch_prober.go:29] interesting pod/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:13:10.000785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:09.993243 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 16:13:10.313435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:10.312467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:10.379777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:10.375940 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:13:10.635958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:10.635924 4573 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:13:10.636137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:10.635973 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 16:13:10.989978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:10.989877 4573 patch_prober.go:29] interesting pod/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:13:10.990565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:10.990531 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 16:13:11.195808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.175836 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:11.635944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.635838 4573 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:13:11.636298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.636202 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 16:13:11.769395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.768978 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:13:11.938516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.938418 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-envoy\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.938516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.938508 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-podinfo\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.938765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.938666 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istiod-ca-cert\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.938860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.938843 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-data\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.938973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.938959 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-kube-api-access-2thrl\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-kube-api-access-2thrl\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.939032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.939001 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-credential-socket\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.939092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.939079 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-socket\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.939148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.939129 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-certs\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.939202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.939163 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-token\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-token\") pod \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\" (UID: \"06c3e3b1-d636-46b7-bd80-4ac313ed0b07\") " Nov 01 16:13:11.940409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.940362 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-data" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:11.940594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.940424 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-credential-socket" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:11.943155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.943127 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-certs" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:11.943333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.943308 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istiod-ca-cert" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:11.943417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.943380 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-socket" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:11.954984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.954948 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-envoy" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:11.955146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.955122 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-kube-api-access-2thrl" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "kube-api-access-2thrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:11.955250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.955230 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-token" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:11.957581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.957553 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-podinfo" pod "06c3e3b1-d636-46b7-bd80-4ac313ed0b07" (UID: "06c3e3b1-d636-46b7-bd80-4ac313ed0b07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:13:11.994776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.990404 4573 patch_prober.go:29] interesting pod/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:13:11.994776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:11.990472 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 16:13:12.042264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042224 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.042264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042266 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2thrl\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-kube-api-access-2thrl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.042495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042308 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.042495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042324 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.042495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042340 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.042495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042352 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.042495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042364 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.042495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042376 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.042495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.042388 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c3e3b1-d636-46b7-bd80-4ac313ed0b07-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.125032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.125004 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:13:12.238977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.238880 4573 generic.go:356] "Generic (PLEG): container finished" podID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerID="19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4" exitCode=0 Nov 01 16:13:12.239144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.239112 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" event={"ID":"787148b3-4c00-43f9-b568-2cd1f994c3d9","Type":"ContainerDied","Data":"19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4"} Nov 01 16:13:12.239190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.239173 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" event={"ID":"787148b3-4c00-43f9-b568-2cd1f994c3d9","Type":"ContainerDied","Data":"8c0cc56e25ab9acee1bcd2b827cc866015173beeeafdd73b8dc175f7fdeaf852"} Nov 01 16:13:12.239236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.239208 4573 scope.go:122] "RemoveContainer" containerID="19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4" Nov 01 16:13:12.239275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.239240 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n" Nov 01 16:13:12.242201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.242177 4573 generic.go:356] "Generic (PLEG): container finished" podID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerID="56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60" exitCode=0 Nov 01 16:13:12.242343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.242287 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" Nov 01 16:13:12.242343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.242307 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" event={"ID":"06c3e3b1-d636-46b7-bd80-4ac313ed0b07","Type":"ContainerDied","Data":"56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60"} Nov 01 16:13:12.242511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.242356 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc" event={"ID":"06c3e3b1-d636-46b7-bd80-4ac313ed0b07","Type":"ContainerDied","Data":"33442464b7b9a1b44923a7a0a0ea5eee014aae635b264d68fd333c59ba59b890"} Nov 01 16:13:12.246548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.246524 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-kube-api-access-4jwh5\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-kube-api-access-4jwh5\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.246787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.246735 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/787148b3-4c00-43f9-b568-2cd1f994c3d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787148b3-4c00-43f9-b568-2cd1f994c3d9-istiod-ca-cert\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.247221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.247207 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-credential-socket\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.247504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.247128 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/787148b3-4c00-43f9-b568-2cd1f994c3d9-istiod-ca-cert" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:13:12.247504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.247447 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-credential-socket" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:12.247694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.247682 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-certs\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.247974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.247953 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-certs" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:12.248079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.247964 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-socket\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.248228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.248212 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-podinfo\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.248349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.248334 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-data\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.248478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.248222 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-socket" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:12.248629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.248616 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-token\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-token\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.248719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.248706 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-envoy\") pod \"787148b3-4c00-43f9-b568-2cd1f994c3d9\" (UID: \"787148b3-4c00-43f9-b568-2cd1f994c3d9\") " Nov 01 16:13:12.248804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.248781 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-data" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:12.249225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.249197 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:12.249675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.249560 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-kube-api-access-4jwh5" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "kube-api-access-4jwh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:12.250128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.250084 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/787148b3-4c00-43f9-b568-2cd1f994c3d9-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.250226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.250133 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.250226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.250150 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.250420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.250163 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.250420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.250258 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.250420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.250294 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jwh5\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-kube-api-access-4jwh5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.253338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.251353 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-token" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:13:12.274794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.271891 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-podinfo" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:13:12.283326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.279041 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-envoy" pod "787148b3-4c00-43f9-b568-2cd1f994c3d9" (UID: "787148b3-4c00-43f9-b568-2cd1f994c3d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:13:12.528126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.286027 4573 scope.go:122] "RemoveContainer" containerID="2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89" Nov 01 16:13:12.528126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.389373 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.528126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.389403 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.528126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.389420 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/787148b3-4c00-43f9-b568-2cd1f994c3d9-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:13:12.545185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.545127 4573 scope.go:122] "RemoveContainer" containerID="19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4" Nov 01 16:13:12.549987 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:13:12.549952 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4\": container with ID starting with 19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4 not found: ID does not exist" containerID="19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4" Nov 01 16:13:12.550139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.549994 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4"} err="failed to get container status \"19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4\": rpc error: code = NotFound desc = could not find container \"19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4\": container with ID starting with 19df58569a413b251676e3ba0173f9fadf49dfd5b69d19cd5dccf184809786b4 not found: ID does not exist" Nov 01 16:13:12.550139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.550012 4573 scope.go:122] "RemoveContainer" containerID="2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89" Nov 01 16:13:12.551071 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:13:12.550514 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89\": container with ID starting with 2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89 not found: ID does not exist" containerID="2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89" Nov 01 16:13:12.551172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.551089 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89"} err="failed to get container status \"2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89\": rpc error: code = NotFound desc = could not find container \"2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89\": container with ID starting with 2203978a5e79aa6befffb0ec4383d0cc4828c7158e3d57d927c840ac21a25d89 not found: ID does not exist" Nov 01 16:13:12.551172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.551112 4573 scope.go:122] "RemoveContainer" containerID="56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60" Nov 01 16:13:12.605793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.590486 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc"] Nov 01 16:13:12.605793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.590551 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-kmsmc"] Nov 01 16:13:12.635686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.635653 4573 patch_prober.go:29] interesting pod/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Nov 01 16:13:12.635889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.635716 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Nov 01 16:13:12.651089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.650228 4573 scope.go:122] "RemoveContainer" containerID="56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60" Nov 01 16:13:12.656335 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:13:12.656296 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60\": container with ID starting with 56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60 not found: ID does not exist" containerID="56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60" Nov 01 16:13:12.656555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.656345 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60"} err="failed to get container status \"56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60\": rpc error: code = NotFound desc = could not find container \"56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60\": container with ID starting with 56c7cf83c4aab2d6abb499dc26902af564e4d0221f38bb9b44fc51dc47848e60 not found: ID does not exist" Nov 01 16:13:12.657629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.657606 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n"] Nov 01 16:13:12.693782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.662415 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-jf99n"] Nov 01 16:13:12.736572 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:13:12.736533 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod787148b3_4c00_43f9_b568_2cd1f994c3d9.slice\": RecentStats: unable to find data in memory cache]" Nov 01 16:13:12.858518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:12.858455 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:13.001602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:13.001567 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:13.002432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:13.002403 4573 patch_prober.go:29] interesting pod/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Nov 01 16:13:13.002546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:13.002458 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Nov 01 16:13:13.087774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:13.087446 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:13.649197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:13.649163 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:13.660478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:13.660448 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:13:13.663300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:13.663267 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:13:13.974835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:13.971272 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:14.008292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.008251 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:13:14.010710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.010685 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:13:14.025365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.025337 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:13:14.300498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.300409 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5"] Nov 01 16:13:14.307590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.307554 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-58b0d1-ms8c5"] Nov 01 16:13:14.320766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.320720 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:14.426775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.423275 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:14.462905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.462873 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" path="/var/lib/kubelet/pods/06c3e3b1-d636-46b7-bd80-4ac313ed0b07/volumes" Nov 01 16:13:14.463658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.463636 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" path="/var/lib/kubelet/pods/787148b3-4c00-43f9-b568-2cd1f994c3d9/volumes" Nov 01 16:13:14.464334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.464312 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141" path="/var/lib/kubelet/pods/dfeeda8f-c8d1-4bc8-87d7-88a38fb4a141/volumes" Nov 01 16:13:14.748609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:14.748578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:15.369902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.369864 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps"] Nov 01 16:13:15.373782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.371105 4573 cpu_manager.go:414] "containerMap: removing container" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" Nov 01 16:13:15.373782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.371132 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" Nov 01 16:13:15.373782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.371148 4573 cpu_manager.go:414] "containerMap: removing container" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" Nov 01 16:13:15.373782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.371156 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" Nov 01 16:13:15.373782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.371404 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" Nov 01 16:13:15.373782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.371425 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c3e3b1-d636-46b7-bd80-4ac313ed0b07" containerName="istio-proxy" Nov 01 16:13:15.381127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.381101 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.386221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.386199 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-99e837-serving-cert\"" Nov 01 16:13:15.423512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.423482 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps"] Nov 01 16:13:15.424241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.424215 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 16:13:15.424378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.424298 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 16:13:15.441138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.441108 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:15.523774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.522607 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-metrics-certs\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.523774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.522815 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.523774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.522877 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.523774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.523247 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm76p\" (UniqueName: \"kubernetes.io/projected/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-kube-api-access-bm76p\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.565777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.565117 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:15.636166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.636079 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-metrics-certs\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.636307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.636213 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.636307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.636264 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.636462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.636369 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm76p\" (UniqueName: \"kubernetes.io/projected/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-kube-api-access-bm76p\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.636462 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:13:15.636421 4573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-serving-cert: secret "maas-api-e2e-inf-b-99e837-serving-cert" not found Nov 01 16:13:15.636565 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:13:15.636493 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls podName:18a0ee2f-1449-4e29-9d6b-084fb968a2dd nodeName:}" failed. No retries permitted until 2026-11-01 16:13:16.136477548 +0000 UTC m=+1533.839643466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls") pod "maas-api-e2e-inf-b-99e837-b79444d9d-njlps" (UID: "18a0ee2f-1449-4e29-9d6b-084fb968a2dd") : secret "maas-api-e2e-inf-b-99e837-serving-cert" not found Nov 01 16:13:15.636939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.636902 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-openshift-service-ca\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.640015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.639993 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-metrics-certs\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.648134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.648094 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm76p\" (UniqueName: \"kubernetes.io/projected/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-kube-api-access-bm76p\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:15.815328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:15.815291 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:16.155424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:16.155380 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:16.184282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:16.183821 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls\") pod \"maas-api-e2e-inf-b-99e837-b79444d9d-njlps\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:16.346485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:16.346449 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:16.352715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:16.352688 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:16.743541 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:13:16.743510 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a0ee2f_1449_4e29_9d6b_084fb968a2dd.slice/crio-7a65d2a6585fa47b5444414e9fc7e7a6c39130307863781f8748b3e035dfee8a WatchSource:0}: Error finding container 7a65d2a6585fa47b5444414e9fc7e7a6c39130307863781f8748b3e035dfee8a: Status 404 returned error can't find the container with id 7a65d2a6585fa47b5444414e9fc7e7a6c39130307863781f8748b3e035dfee8a Nov 01 16:13:16.744375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:16.744354 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps"] Nov 01 16:13:16.745165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:16.745146 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:13:17.048539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:17.048507 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:17.338832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:17.336373 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" event={"ID":"18a0ee2f-1449-4e29-9d6b-084fb968a2dd","Type":"ContainerStarted","Data":"7a65d2a6585fa47b5444414e9fc7e7a6c39130307863781f8748b3e035dfee8a"} Nov 01 16:13:17.697542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:17.697506 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:18.478858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:18.478825 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:19.147133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:19.147101 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:19.368130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:19.368089 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" event={"ID":"18a0ee2f-1449-4e29-9d6b-084fb968a2dd","Type":"ContainerStarted","Data":"f33a37bc5e082892fc94a04a3de30d2768b1cf63df4edc39bb521a705570be33"} Nov 01 16:13:19.368327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:19.368184 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:19.385670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:19.385617 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" podStartSLOduration=2.663533164 podStartE2EDuration="4.385603772s" totalImagesPullingTime="1.722070608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:19.384785375 +0000 UTC m=+1537.087951312" watchObservedRunningTime="2026-11-01 16:13:19.385603772 +0000 UTC m=+1537.088769707" Nov 01 16:13:19.816509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:19.816472 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:20.518050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:20.517999 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:20.973917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:20.973883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:21.088061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:21.088013 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:21.288979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:21.288900 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:21.393773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:21.390401 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:22.070266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:22.070235 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:22.201774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:22.188740 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:22.546848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:22.546809 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:22.715777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:22.713723 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:23.265671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:23.265603 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:24.023005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.022971 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk"] Nov 01 16:13:24.028425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.028398 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8626424e-9plkk"] Nov 01 16:13:24.119138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.119103 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47"] Nov 01 16:13:24.119933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.119905 4573 cpu_manager.go:414] "containerMap: removing container" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" Nov 01 16:13:24.119933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.119933 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" Nov 01 16:13:24.120060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.120043 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="787148b3-4c00-43f9-b568-2cd1f994c3d9" containerName="istio-proxy" Nov 01 16:13:24.122986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.122964 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.129976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.129943 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-10b00a-serving-cert\"" Nov 01 16:13:24.155808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.155407 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47"] Nov 01 16:13:24.163361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.163297 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:24.202226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.202192 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24z6j\" (UniqueName: \"kubernetes.io/projected/d07785d5-56d1-4261-b44e-3397f98d9cc9-kube-api-access-24z6j\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.202542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.202515 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-maas-api-tls\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.202691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.202669 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07785d5-56d1-4261-b44e-3397f98d9cc9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.202825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.202805 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-metrics-certs\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.264520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.264475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:24.320826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.314131 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-metrics-certs\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.320826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.314276 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24z6j\" (UniqueName: \"kubernetes.io/projected/d07785d5-56d1-4261-b44e-3397f98d9cc9-kube-api-access-24z6j\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.320826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.314556 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-maas-api-tls\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.320826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.314651 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07785d5-56d1-4261-b44e-3397f98d9cc9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.320826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.315817 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07785d5-56d1-4261-b44e-3397f98d9cc9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.349773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.344376 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-metrics-certs\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.361564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.361523 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-maas-api-tls\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.369490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.369427 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24z6j\" (UniqueName: \"kubernetes.io/projected/d07785d5-56d1-4261-b44e-3397f98d9cc9-kube-api-access-24z6j\") pod \"maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.372662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.370707 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:24.453812 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:13:24.450790 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:13:24.470517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.470484 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04059156-67c1-42ac-8c69-b0298c6c5c6f" path="/var/lib/kubelet/pods/04059156-67c1-42ac-8c69-b0298c6c5c6f/volumes" Nov 01 16:13:24.473538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.473296 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:24.882323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:24.876438 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47"] Nov 01 16:13:25.169828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:25.169792 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:25.384198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:25.384177 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:13:25.470916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:25.467861 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" event={"ID":"d07785d5-56d1-4261-b44e-3397f98d9cc9","Type":"ContainerStarted","Data":"d59ea225ac276b06bba7850b31446cde3f42f16b30f98fda5bfd9002c2536784"} Nov 01 16:13:25.470916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:25.467935 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" event={"ID":"d07785d5-56d1-4261-b44e-3397f98d9cc9","Type":"ContainerStarted","Data":"3aede44e1922dabc50a968a9cc35f8757ae7841f86b761c8e61b9b71338a6109"} Nov 01 16:13:25.470916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:25.468128 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:25.489398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:25.489348 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" podStartSLOduration=1.0530881029999999 podStartE2EDuration="1.489333017s" totalImagesPullingTime="436.244914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:25.484874574 +0000 UTC m=+1543.188040514" watchObservedRunningTime="2026-11-01 16:13:25.489333017 +0000 UTC m=+1543.192498953" Nov 01 16:13:26.136410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:26.136375 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:26.586128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:26.586053 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:26.770766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:26.770711 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:27.442984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:27.442946 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:27.747370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:27.747287 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:27.847709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:27.847677 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:28.575572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:28.575531 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:29.475843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:29.475729 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:30.118501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:30.118473 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:30.781333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:30.781300 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:31.380379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:31.380318 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:31.480385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:31.480349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:31.498941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:31.498917 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:13:32.166774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:32.162898 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:32.878398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:32.878359 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:33.562487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:33.562422 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:34.200464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:34.200429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:34.876352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:34.876321 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:35.432188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:35.432155 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:35.768826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:35.767578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:35.918196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:35.918152 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:36.063788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:36.063699 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm"] Nov 01 16:13:36.068794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:36.068763 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26310f-lt6dm"] Nov 01 16:13:36.236675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:36.236636 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:36.321686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:36.321613 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:36.484777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:36.483927 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90921542-85ee-45b1-ab3f-f257e1ff5c06" path="/var/lib/kubelet/pods/90921542-85ee-45b1-ab3f-f257e1ff5c06/volumes" Nov 01 16:13:36.661894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:36.661859 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:37.057423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:37.057389 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:37.786135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:37.786099 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:38.321483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:38.321449 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:38.995995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:38.995959 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:39.681758 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:39.681714 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:40.467312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:40.467282 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:41.218422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:41.218385 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:41.979882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:41.979847 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:42.561704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:42.561674 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:42.879364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:42.879311 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:43.698334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:43.698293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:43.808847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:43.806270 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:44.031567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:44.031479 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:44.673124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:44.673088 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:44.834083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:44.834042 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:45.250219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:45.250181 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:45.408775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:45.408085 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:45.780859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:45.780812 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:46.024981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:46.024945 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7"] Nov 01 16:13:46.030106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:46.030071 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1139f235-cxsb7"] Nov 01 16:13:46.460892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:46.460858 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2246fdf6-6d78-4217-a4cc-c9b36391c867" path="/var/lib/kubelet/pods/2246fdf6-6d78-4217-a4cc-c9b36391c867/volumes" Nov 01 16:13:46.643378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:46.643341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:47.256946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:47.256903 4573 scope.go:122] "RemoveContainer" containerID="a114be7b9d0daccd0aadbb6a7679e01a8e8ce5d69d9acbeb01cb53cdbae366a8" Nov 01 16:13:47.746384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:47.746344 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:48.962323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:48.962292 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:49.169890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:49.169852 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:49.419776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:49.409083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:50.426788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:50.424867 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:50.495773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:50.453901 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:50.535733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:50.532853 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:50.685824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:50.682881 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 16:13:50.843218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:50.843179 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:50.960008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:50.959919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:51.234823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.216845 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw"] Nov 01 16:13:51.248242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.241963 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw"] Nov 01 16:13:51.248242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.242100 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:51.261821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.260082 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-9e044c\"" Nov 01 16:13:51.261821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.260220 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-9e044c-dockercfg-b44jc\"" Nov 01 16:13:51.296863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.296792 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:51.392420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.392375 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9zd\" (UniqueName: \"kubernetes.io/projected/88acad1c-61c1-4887-8abc-767ccf45e509-kube-api-access-tf9zd\") pod \"payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw\" (UID: \"88acad1c-61c1-4887-8abc-767ccf45e509\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:51.393054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.393002 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acad1c-61c1-4887-8abc-767ccf45e509-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw\" (UID: \"88acad1c-61c1-4887-8abc-767ccf45e509\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:51.512825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.511961 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9zd\" (UniqueName: \"kubernetes.io/projected/88acad1c-61c1-4887-8abc-767ccf45e509-kube-api-access-tf9zd\") pod \"payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw\" (UID: \"88acad1c-61c1-4887-8abc-767ccf45e509\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:51.512825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.512270 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acad1c-61c1-4887-8abc-767ccf45e509-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw\" (UID: \"88acad1c-61c1-4887-8abc-767ccf45e509\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:51.569544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.569508 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9zd\" (UniqueName: \"kubernetes.io/projected/88acad1c-61c1-4887-8abc-767ccf45e509-kube-api-access-tf9zd\") pod \"payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw\" (UID: \"88acad1c-61c1-4887-8abc-767ccf45e509\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:51.590824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.590444 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acad1c-61c1-4887-8abc-767ccf45e509-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw\" (UID: \"88acad1c-61c1-4887-8abc-767ccf45e509\") " pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:51.590824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.590772 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:51.936737 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:13:51.936009 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88acad1c_61c1_4887_8abc_767ccf45e509.slice/crio-e811ea7fa4ddf0c0230084a0f01d072d00029a9ddfe1ea4d0b9737a0f9315f37 WatchSource:0}: Error finding container e811ea7fa4ddf0c0230084a0f01d072d00029a9ddfe1ea4d0b9737a0f9315f37: Status 404 returned error can't find the container with id e811ea7fa4ddf0c0230084a0f01d072d00029a9ddfe1ea4d0b9737a0f9315f37 Nov 01 16:13:51.936737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:51.936416 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw"] Nov 01 16:13:52.006282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:52.006248 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" event={"ID":"88acad1c-61c1-4887-8abc-767ccf45e509","Type":"ContainerStarted","Data":"e811ea7fa4ddf0c0230084a0f01d072d00029a9ddfe1ea4d0b9737a0f9315f37"} Nov 01 16:13:52.264948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:52.264874 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:53.015740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:53.015696 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" event={"ID":"88acad1c-61c1-4887-8abc-767ccf45e509","Type":"ContainerStarted","Data":"ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d"} Nov 01 16:13:53.016150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:53.015814 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:13:53.037713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:53.037660 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" podStartSLOduration=3.037647864 podStartE2EDuration="3.037647864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:13:53.033182805 +0000 UTC m=+1570.736348747" watchObservedRunningTime="2026-11-01 16:13:53.037647864 +0000 UTC m=+1570.740813800" Nov 01 16:13:53.389380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:53.389342 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:54.252653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:54.252620 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:54.366938 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:13:54.366901 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:13:55.130809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:55.128349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:55.426107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:55.426030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:56.083386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:56.083348 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:56.879619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:56.879580 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:57.686517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:57.686487 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:58.466557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:58.466521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:59.087615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:59.087584 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:13:59.851592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:13:59.851559 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:00.715583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:00.715550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:00.817721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:00.817679 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:01.661585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:01.661549 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:02.380121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:02.380090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:03.092203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:03.092169 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:03.890668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:03.890621 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:04.026820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:04.026743 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:14:04.711576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:04.711544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:05.356598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:05.356565 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:05.996354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:05.996327 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:06.102510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:06.102450 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:06.779851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:06.779820 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:07.366886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:07.366850 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:07.766114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:07.766035 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:08.547837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:08.547806 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:09.324629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:09.324595 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:10.054621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:10.054586 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:10.686583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:10.686540 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:10.802164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:10.802125 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:11.571583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:11.571550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:12.498419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:12.498368 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:12.632938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:12.632900 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:13.355674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:13.355639 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:14.061812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:14.056910 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5"] Nov 01 16:14:14.100780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:14.081631 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cfe49e0-26wg5"] Nov 01 16:14:14.311733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:14.311699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:14.465601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:14.465562 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c031fe-ac69-4538-a859-ee38f0837757" path="/var/lib/kubelet/pods/59c031fe-ac69-4538-a859-ee38f0837757/volumes" Nov 01 16:14:15.130043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:15.130003 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:15.945619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:15.945589 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:16.739827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:16.739796 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:17.330804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:17.330769 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:17.569808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:17.569766 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:17.748036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:17.747996 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:18.177655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:18.177609 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:18.285761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:18.285706 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:18.379963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:18.379929 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:18.818094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:18.818058 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:19.504030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:19.503992 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:20.436322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:20.436289 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:21.237979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:21.237938 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:22.063866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:22.063829 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:22.779172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:22.779137 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:23.468555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:23.468527 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:24.159485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:24.159452 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:24.369291 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:14:24.369252 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:14:24.944354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:24.944324 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:25.656330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:25.656297 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:26.346549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:26.346507 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:27.083142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:27.083108 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:27.933020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:27.932986 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:28.705125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:28.705089 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:29.460366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:29.460338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:30.199563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:30.199530 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:30.942807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:30.942769 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:31.427645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:31.427610 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:31.778354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:31.778274 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:32.463871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:32.463841 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:33.191503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:33.191468 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:33.875581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:33.875546 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:34.675994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:34.675956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:35.498074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:35.498040 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:36.220823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:36.220791 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:37.296542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:37.296503 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:38.149882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:38.149837 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:38.772076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:38.772046 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:39.523949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:39.523911 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:40.167264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:40.167227 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:40.795562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:40.795486 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:41.534884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:41.534852 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:42.360554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:42.360523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:43.026806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:43.026766 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:43.796567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:43.796533 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:44.601570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:44.601527 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:45.269997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:45.269941 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:45.462994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:45.462959 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:45.561399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:45.561317 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:46.027907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:46.027871 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:46.798960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:46.798922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:47.312091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:47.312061 4573 scope.go:122] "RemoveContainer" containerID="42ad5ff8dfae61a8a0e65f915fd8b1add4c7dc1778694f01af0b20b522ee0eb2" Nov 01 16:14:47.327025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:47.327000 4573 scope.go:122] "RemoveContainer" containerID="87ddfdf5039620688e87742981ebecff2a933f044c680aff86ba9c3fd73c6ee3" Nov 01 16:14:47.338620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:47.338593 4573 scope.go:122] "RemoveContainer" containerID="4d1d0ff6f7ce9bfe2f74d228394ee3d23190e3b6bd5ddfe19ca328e0805606fa" Nov 01 16:14:47.551428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:47.551396 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:48.190550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:48.190512 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:48.940027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:48.939994 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:49.750851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:49.750814 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:50.372116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:50.372083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:51.081822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:51.081790 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:51.403129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:51.403097 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:14:51.981742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:51.981689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:52.848775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:52.848695 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:54.434353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:54.434187 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:54.658520 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:14:54.658390 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:14:55.296689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:55.296658 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:56.091496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:56.091393 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:56.834809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:56.834775 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:57.648275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:57.648227 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:58.404328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:58.404291 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:59.187826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:59.187731 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:14:59.990726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:14:59.990685 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:00.166026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.165993 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn"] Nov 01 16:15:00.175850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.175819 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:00.188116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.188084 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:15:00.189300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.188644 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2"] Nov 01 16:15:00.196429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.196407 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:00.234639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.234608 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn"] Nov 01 16:15:00.259771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.256340 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2"] Nov 01 16:15:00.290973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.290437 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nczrl\" (UniqueName: \"kubernetes.io/projected/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-kube-api-access-nczrl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn\" (UID: \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:00.290973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.290546 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn\" (UID: \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:00.309243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.309195 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f"] Nov 01 16:15:00.310630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.309574 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" containerID="cri-o://54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6" gracePeriod=30 Nov 01 16:15:00.392635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.392601 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nczrl\" (UniqueName: \"kubernetes.io/projected/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-kube-api-access-nczrl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn\" (UID: \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:00.392834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.392680 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn\" (UID: \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:00.392888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.392831 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvd2\" (UniqueName: \"kubernetes.io/projected/3b6bb026-2955-4763-aa43-299714930a67-kube-api-access-wmvd2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2\" (UID: \"3b6bb026-2955-4763-aa43-299714930a67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:00.392941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.392893 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6bb026-2955-4763-aa43-299714930a67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2\" (UID: \"3b6bb026-2955-4763-aa43-299714930a67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:00.393525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.393499 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn\" (UID: \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:00.405845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.405819 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nczrl\" (UniqueName: \"kubernetes.io/projected/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-kube-api-access-nczrl\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn\" (UID: \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:00.495279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.495240 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvd2\" (UniqueName: \"kubernetes.io/projected/3b6bb026-2955-4763-aa43-299714930a67-kube-api-access-wmvd2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2\" (UID: \"3b6bb026-2955-4763-aa43-299714930a67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:00.495450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.495331 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6bb026-2955-4763-aa43-299714930a67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2\" (UID: \"3b6bb026-2955-4763-aa43-299714930a67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:00.496031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.496010 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6bb026-2955-4763-aa43-299714930a67-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2\" (UID: \"3b6bb026-2955-4763-aa43-299714930a67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:00.503127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.503062 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvd2\" (UniqueName: \"kubernetes.io/projected/3b6bb026-2955-4763-aa43-299714930a67-kube-api-access-wmvd2\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2\" (UID: \"3b6bb026-2955-4763-aa43-299714930a67\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:00.521989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.521945 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:00.557651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.557611 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7"] Nov 01 16:15:00.558131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.558101 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" containerID="cri-o://c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8" gracePeriod=30 Nov 01 16:15:00.560425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.560220 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:00.874563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.874525 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn"] Nov 01 16:15:00.876629 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:15:00.876598 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc8b9da_6b7b_4854_b6ac_6da05ec6a8b6.slice/crio-056e689c9d4c506b9eaa5117bfc014d27825af0d406d55a3fd735e83d21aedd7 WatchSource:0}: Error finding container 056e689c9d4c506b9eaa5117bfc014d27825af0d406d55a3fd735e83d21aedd7: Status 404 returned error can't find the container with id 056e689c9d4c506b9eaa5117bfc014d27825af0d406d55a3fd735e83d21aedd7 Nov 01 16:15:00.910727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.910695 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:00.932023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:00.931991 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2"] Nov 01 16:15:00.932652 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:15:00.932622 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6bb026_2955_4763_aa43_299714930a67.slice/crio-082811b60bdae37f2fd85625f611454624606852717e81d2493be1643b22390c WatchSource:0}: Error finding container 082811b60bdae37f2fd85625f611454624606852717e81d2493be1643b22390c: Status 404 returned error can't find the container with id 082811b60bdae37f2fd85625f611454624606852717e81d2493be1643b22390c Nov 01 16:15:01.153158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:01.153067 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" event={"ID":"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6","Type":"ContainerStarted","Data":"056e689c9d4c506b9eaa5117bfc014d27825af0d406d55a3fd735e83d21aedd7"} Nov 01 16:15:01.154525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:01.154496 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" event={"ID":"3b6bb026-2955-4763-aa43-299714930a67","Type":"ContainerStarted","Data":"082811b60bdae37f2fd85625f611454624606852717e81d2493be1643b22390c"} Nov 01 16:15:01.693549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:01.693513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:02.166171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:02.166137 4573 generic.go:356] "Generic (PLEG): container finished" podID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerID="dbc557e54d3b4bcd7256bd173dffa96faae7b762be99e4c4901773ec88bd124f" exitCode=60 Nov 01 16:15:02.166356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:02.166225 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" event={"ID":"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6","Type":"ContainerDied","Data":"dbc557e54d3b4bcd7256bd173dffa96faae7b762be99e4c4901773ec88bd124f"} Nov 01 16:15:02.166624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:02.166598 4573 scope.go:122] "RemoveContainer" containerID="dbc557e54d3b4bcd7256bd173dffa96faae7b762be99e4c4901773ec88bd124f" Nov 01 16:15:02.168344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:02.168321 4573 generic.go:356] "Generic (PLEG): container finished" podID="3b6bb026-2955-4763-aa43-299714930a67" containerID="b41b0a8a33fe662e22a51ab25b4ec3c217e090a83a4c23a8c1bfc8f32442bbae" exitCode=60 Nov 01 16:15:02.168447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:02.168388 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" event={"ID":"3b6bb026-2955-4763-aa43-299714930a67","Type":"ContainerDied","Data":"b41b0a8a33fe662e22a51ab25b4ec3c217e090a83a4c23a8c1bfc8f32442bbae"} Nov 01 16:15:02.168665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:02.168642 4573 scope.go:122] "RemoveContainer" containerID="b41b0a8a33fe662e22a51ab25b4ec3c217e090a83a4c23a8c1bfc8f32442bbae" Nov 01 16:15:02.456180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:02.456143 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:03.183182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:03.183146 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" event={"ID":"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6","Type":"ContainerStarted","Data":"f2455be3f40d66bab8b915a2b0da7420ffb7f5739674ce12c9d73197c01f8e84"} Nov 01 16:15:03.183562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:03.183510 4573 scope.go:122] "RemoveContainer" containerID="f2455be3f40d66bab8b915a2b0da7420ffb7f5739674ce12c9d73197c01f8e84" Nov 01 16:15:03.184012 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:03.183987 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn_odh-ai-gateway-infra(5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" Nov 01 16:15:03.185430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:03.185404 4573 generic.go:356] "Generic (PLEG): container finished" podID="3b6bb026-2955-4763-aa43-299714930a67" containerID="1d59a136c21236aa578f8637de56246c2f89aa320784231b08adbeba70eb9a20" exitCode=60 Nov 01 16:15:03.185507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:03.185468 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" event={"ID":"3b6bb026-2955-4763-aa43-299714930a67","Type":"ContainerDied","Data":"1d59a136c21236aa578f8637de56246c2f89aa320784231b08adbeba70eb9a20"} Nov 01 16:15:03.185560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:03.185528 4573 scope.go:122] "RemoveContainer" containerID="b41b0a8a33fe662e22a51ab25b4ec3c217e090a83a4c23a8c1bfc8f32442bbae" Nov 01 16:15:03.185923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:03.185905 4573 scope.go:122] "RemoveContainer" containerID="1d59a136c21236aa578f8637de56246c2f89aa320784231b08adbeba70eb9a20" Nov 01 16:15:03.186315 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:03.186295 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2_odh-ai-gateway-infra(3b6bb026-2955-4763-aa43-299714930a67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" podUID="3b6bb026-2955-4763-aa43-299714930a67" Nov 01 16:15:03.463117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:03.463079 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:04.011947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.011900 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:15:04.012136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.012000 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:15:04.197763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.197723 4573 generic.go:356] "Generic (PLEG): container finished" podID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerID="f2455be3f40d66bab8b915a2b0da7420ffb7f5739674ce12c9d73197c01f8e84" exitCode=60 Nov 01 16:15:04.200433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.197801 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" event={"ID":"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6","Type":"ContainerDied","Data":"f2455be3f40d66bab8b915a2b0da7420ffb7f5739674ce12c9d73197c01f8e84"} Nov 01 16:15:04.200433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.197867 4573 scope.go:122] "RemoveContainer" containerID="dbc557e54d3b4bcd7256bd173dffa96faae7b762be99e4c4901773ec88bd124f" Nov 01 16:15:04.200433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.198199 4573 scope.go:122] "RemoveContainer" containerID="f2455be3f40d66bab8b915a2b0da7420ffb7f5739674ce12c9d73197c01f8e84" Nov 01 16:15:04.200433 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:04.198594 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn_odh-ai-gateway-infra(5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" Nov 01 16:15:04.200433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.200026 4573 scope.go:122] "RemoveContainer" containerID="1d59a136c21236aa578f8637de56246c2f89aa320784231b08adbeba70eb9a20" Nov 01 16:15:04.200433 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:04.200381 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2_odh-ai-gateway-infra(3b6bb026-2955-4763-aa43-299714930a67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" podUID="3b6bb026-2955-4763-aa43-299714930a67" Nov 01 16:15:04.239216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.221652 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:04.922345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:04.922300 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:05.210563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.210480 4573 generic.go:356] "Generic (PLEG): container finished" podID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerID="6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d" exitCode=0 Nov 01 16:15:05.210981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.210552 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" event={"ID":"9c921e79-5128-4b20-bdc3-302e60b3e5d6","Type":"ContainerDied","Data":"6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d"} Nov 01 16:15:05.211114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.211090 4573 scope.go:122] "RemoveContainer" containerID="6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d" Nov 01 16:15:05.212886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.212866 4573 scope.go:122] "RemoveContainer" containerID="f2455be3f40d66bab8b915a2b0da7420ffb7f5739674ce12c9d73197c01f8e84" Nov 01 16:15:05.212992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.212963 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:05.213065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.213035 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:05.213123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.213069 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:05.213123 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:05.213096 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn_odh-ai-gateway-infra(5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" Nov 01 16:15:05.703337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.703284 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:05.704675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.704643 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 16:15:05.741807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.741694 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62"] Nov 01 16:15:05.742640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.742615 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" Nov 01 16:15:05.742640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.742636 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" Nov 01 16:15:05.742908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.742664 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" Nov 01 16:15:05.742908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.742669 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" Nov 01 16:15:05.742908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.742817 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" Nov 01 16:15:05.742908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.742831 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5747c189-9aad-474e-b650-79061531f01d" containerName="istio-proxy" Nov 01 16:15:05.774727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.774646 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.775841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.774550 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62"] Nov 01 16:15:05.780185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.780158 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-99afca-openshift-default-dockercfg-7gj25\"" Nov 01 16:15:05.823066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823029 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-certs\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823159 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-credential-socket\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823269 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-data\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823323 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5747c189-9aad-474e-b650-79061531f01d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747c189-9aad-474e-b650-79061531f01d-istio-podinfo\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823434 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-envoy\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823587 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-kube-api-access-6zpfq\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-kube-api-access-6zpfq\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823628 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-istio-token\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-istio-token\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823711 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-socket\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823791 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5747c189-9aad-474e-b650-79061531f01d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747c189-9aad-474e-b650-79061531f01d-istiod-ca-cert\") pod \"5747c189-9aad-474e-b650-79061531f01d\" (UID: \"5747c189-9aad-474e-b650-79061531f01d\") " Nov 01 16:15:05.823950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.823909 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-credential-socket" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.824095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.824072 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e67049-8316-4cef-8d25-3e09e68797fe-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.824186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.824123 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.824278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.824251 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e67049-8316-4cef-8d25-3e09e68797fe-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.824448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.824418 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.824743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.824607 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.824857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.824842 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.825294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.824947 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.825294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.825010 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st6nq\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-kube-api-access-st6nq\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.825294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.825042 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.825294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.825067 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5747c189-9aad-474e-b650-79061531f01d-istiod-ca-cert" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:05.825566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.825357 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5747c189-9aad-474e-b650-79061531f01d-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.825566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.825380 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.825777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.825731 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-socket" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.825988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.825968 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-data" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.826187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.826164 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-certs" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.897566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.887781 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-envoy" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:05.897566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.891854 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-kube-api-access-6zpfq" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "kube-api-access-6zpfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:05.927804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.922980 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5747c189-9aad-474e-b650-79061531f01d-istio-podinfo" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.929536 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.929619 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st6nq\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-kube-api-access-st6nq\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.929652 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.929787 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e67049-8316-4cef-8d25-3e09e68797fe-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.929818 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.929861 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e67049-8316-4cef-8d25-3e09e68797fe-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.929936 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.930086 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.930272 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.930418 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.930435 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.930451 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.930464 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5747c189-9aad-474e-b650-79061531f01d-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.930480 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5747c189-9aad-474e-b650-79061531f01d-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.933833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.930495 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zpfq\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-kube-api-access-6zpfq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:05.945182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.935565 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-credential-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.945182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.937052 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-certs\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.945182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.937689 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e67049-8316-4cef-8d25-3e09e68797fe-istiod-ca-cert\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.945182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.938341 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-data\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.945182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.938550 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-socket\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:05.995557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:05.992461 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 16:15:06.015623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.006343 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e67049-8316-4cef-8d25-3e09e68797fe-istio-podinfo\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:06.020892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.020855 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-istio-token" pod "5747c189-9aad-474e-b650-79061531f01d" (UID: "5747c189-9aad-474e-b650-79061531f01d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.024478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.024412 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-envoy\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.034338 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-kube-api-access-xg96w\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-kube-api-access-xg96w\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.034386 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-podinfo\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.034408 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-credential-socket\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.034485 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-socket\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.034507 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-certs\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.034563 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-data\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.034953 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-envoy\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.034977 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-data" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.035014 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36b9a4c9-737d-4373-b1ee-aec239b0803e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9a4c9-737d-4373-b1ee-aec239b0803e-istiod-ca-cert\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.035083 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-token\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-token\") pod \"36b9a4c9-737d-4373-b1ee-aec239b0803e\" (UID: \"36b9a4c9-737d-4373-b1ee-aec239b0803e\") " Nov 01 16:15:06.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.035386 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st6nq\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-kube-api-access-st6nq\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:06.037805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.036052 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5747c189-9aad-474e-b650-79061531f01d-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.037805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.036076 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.037805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.036414 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36b9a4c9-737d-4373-b1ee-aec239b0803e-istiod-ca-cert" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:06.037805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.036783 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-socket" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.037805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.036849 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm"] Nov 01 16:15:06.037805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.036983 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-credential-socket" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.037805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.037374 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-istio-token\") pod \"e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:06.038203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.038122 4573 cpu_manager.go:414] "containerMap: removing container" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" Nov 01 16:15:06.038203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.038141 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" Nov 01 16:15:06.039542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.038315 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-podinfo" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:15:06.039542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.038369 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerName="istio-proxy" Nov 01 16:15:06.039542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.038894 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-certs" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.042150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.042104 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-kube-api-access-xg96w" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "kube-api-access-xg96w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.042263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.042218 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-token" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:06.043440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.043420 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.044352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.044318 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-envoy" pod "36b9a4c9-737d-4373-b1ee-aec239b0803e" (UID: "36b9a4c9-737d-4373-b1ee-aec239b0803e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:15:06.068845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.068817 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0c091d-openshift-default-dockercfg-c5fg8\"" Nov 01 16:15:06.073212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.073150 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm"] Nov 01 16:15:06.115467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.115183 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:06.143938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.143890 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-credential-socket\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.144247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.144222 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kccrj\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-kube-api-access-kccrj\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.144400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.144388 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febdd5f-7998-4501-a41d-d87ef58a3703-istiod-ca-cert\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.144573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.144561 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-podinfo\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.144927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.144902 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-envoy\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.145104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145087 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-socket\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.145236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145223 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-data\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.145525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145465 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-certs\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.145610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145573 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-token\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.145857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145839 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg96w\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-kube-api-access-xg96w\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.146021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145861 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.146021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145879 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.146021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145897 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.146021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145910 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.146021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145925 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.146021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145937 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36b9a4c9-737d-4373-b1ee-aec239b0803e-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.146021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.145951 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36b9a4c9-737d-4373-b1ee-aec239b0803e-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:06.228314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.225597 4573 generic.go:356] "Generic (PLEG): container finished" podID="36b9a4c9-737d-4373-b1ee-aec239b0803e" containerID="c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8" exitCode=0 Nov 01 16:15:06.228314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.225663 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" event={"ID":"36b9a4c9-737d-4373-b1ee-aec239b0803e","Type":"ContainerDied","Data":"c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8"} Nov 01 16:15:06.228314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.225718 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" event={"ID":"36b9a4c9-737d-4373-b1ee-aec239b0803e","Type":"ContainerDied","Data":"fdb4b04e0df25defb8f7e1bf8c38a527a62bd32027dc24561d49e35675d4e13a"} Nov 01 16:15:06.228314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.225759 4573 scope.go:122] "RemoveContainer" containerID="c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8" Nov 01 16:15:06.228314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.225868 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7" Nov 01 16:15:06.232979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.230578 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" event={"ID":"9c921e79-5128-4b20-bdc3-302e60b3e5d6","Type":"ContainerStarted","Data":"262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad"} Nov 01 16:15:06.252826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.247810 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-token\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.252826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.247997 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-credential-socket\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.252826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.248032 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kccrj\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-kube-api-access-kccrj\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.252826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.248061 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febdd5f-7998-4501-a41d-d87ef58a3703-istiod-ca-cert\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.252826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.248962 4573 generic.go:356] "Generic (PLEG): container finished" podID="5747c189-9aad-474e-b650-79061531f01d" containerID="54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6" exitCode=0 Nov 01 16:15:06.252826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.249055 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" event={"ID":"5747c189-9aad-474e-b650-79061531f01d","Type":"ContainerDied","Data":"54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6"} Nov 01 16:15:06.252826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.249097 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" event={"ID":"5747c189-9aad-474e-b650-79061531f01d","Type":"ContainerDied","Data":"78c7b2091756a764e493ec547426e823c6647d99d705e10923ddece3401a18d9"} Nov 01 16:15:06.252826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.249169 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f" Nov 01 16:15:06.256188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.252997 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-credential-socket\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.256188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.253858 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-podinfo\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.256188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.253909 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-envoy\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.256188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.253938 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-socket\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.261557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.261151 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-data\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.261557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.261329 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-certs\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.266489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.265395 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-data\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.266489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.266176 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-socket\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.268744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.268169 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kccrj\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-kube-api-access-kccrj\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.268744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.268308 4573 scope.go:122] "RemoveContainer" containerID="c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8" Nov 01 16:15:06.268744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.268461 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-certs\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.271838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.271811 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febdd5f-7998-4501-a41d-d87ef58a3703-istiod-ca-cert\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.273577 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:06.273433 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8\": container with ID starting with c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8 not found: ID does not exist" containerID="c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8" Nov 01 16:15:06.273826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.273741 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8"} err="failed to get container status \"c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8\": rpc error: code = NotFound desc = could not find container \"c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8\": container with ID starting with c5fbc8a52ac663bbd93e39d9e9a859af1ad84fea685055a9da24049a312739e8 not found: ID does not exist" Nov 01 16:15:06.274167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.274152 4573 scope.go:122] "RemoveContainer" containerID="54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6" Nov 01 16:15:06.326170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.326150 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7"] Nov 01 16:15:06.333269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.333196 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-5zvm7"] Nov 01 16:15:06.346914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.346844 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f"] Nov 01 16:15:06.348586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.348560 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-token\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.349172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.349148 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-podinfo\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.351483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.351461 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-envoy\") pod \"e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.352641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.352619 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-jjk7f"] Nov 01 16:15:06.355735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.355711 4573 scope.go:122] "RemoveContainer" containerID="2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c" Nov 01 16:15:06.420091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.420060 4573 scope.go:122] "RemoveContainer" containerID="54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6" Nov 01 16:15:06.420557 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:06.420505 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6\": container with ID starting with 54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6 not found: ID does not exist" containerID="54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6" Nov 01 16:15:06.420667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.420546 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6"} err="failed to get container status \"54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6\": rpc error: code = NotFound desc = could not find container \"54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6\": container with ID starting with 54bd79a7c774d1f1c48138f89e14659f10233b852052186f58233a1dfbe329f6 not found: ID does not exist" Nov 01 16:15:06.420667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.420573 4573 scope.go:122] "RemoveContainer" containerID="2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c" Nov 01 16:15:06.421531 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:06.421502 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c\": container with ID starting with 2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c not found: ID does not exist" containerID="2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c" Nov 01 16:15:06.421646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.421539 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c"} err="failed to get container status \"2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c\": rpc error: code = NotFound desc = could not find container \"2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c\": container with ID starting with 2b687a9423aaff18f7e72af1a8a6554263f4b05620cb2b930875e8456f7d798c not found: ID does not exist" Nov 01 16:15:06.436032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.436003 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:06.467247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.467209 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36b9a4c9-737d-4373-b1ee-aec239b0803e" path="/var/lib/kubelet/pods/36b9a4c9-737d-4373-b1ee-aec239b0803e/volumes" Nov 01 16:15:06.468319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.468291 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5747c189-9aad-474e-b650-79061531f01d" path="/var/lib/kubelet/pods/5747c189-9aad-474e-b650-79061531f01d/volumes" Nov 01 16:15:06.550952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.538702 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62"] Nov 01 16:15:06.550952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.541567 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.550952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.541705 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.550952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.541810 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.705743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.705689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:06.937467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.937435 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm"] Nov 01 16:15:06.938318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.938284 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.938423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.938354 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:06.938423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:06.938383 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:15:07.225336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.225242 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:15:07.225336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.225278 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:15:07.225580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.225327 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 16:15:07.281772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.281362 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" event={"ID":"1febdd5f-7998-4501-a41d-d87ef58a3703","Type":"ContainerStarted","Data":"dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046"} Nov 01 16:15:07.281772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.281432 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" event={"ID":"1febdd5f-7998-4501-a41d-d87ef58a3703","Type":"ContainerStarted","Data":"1565520f9668caabe0d75f1cab5942a35f50940bbbf87af818c09f11946e3757"} Nov 01 16:15:07.294291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.294253 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" event={"ID":"84e67049-8316-4cef-8d25-3e09e68797fe","Type":"ContainerStarted","Data":"3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22"} Nov 01 16:15:07.294565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.294543 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" event={"ID":"84e67049-8316-4cef-8d25-3e09e68797fe","Type":"ContainerStarted","Data":"6131bcd7ab8a960d5cb8dc6b21ad34b348c7181091d7162f141df6f77598b93f"} Nov 01 16:15:07.309237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.309177 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" podStartSLOduration=49.309158728 podStartE2EDuration="49.309158728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:07.305787889 +0000 UTC m=+1645.008953824" watchObservedRunningTime="2026-11-01 16:15:07.309158728 +0000 UTC m=+1645.012324665" Nov 01 16:15:07.340041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.339984 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podStartSLOduration=66.339968801 podStartE2EDuration="1m6.339968801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:14:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:07.33069151 +0000 UTC m=+1645.033857447" watchObservedRunningTime="2026-11-01 16:15:07.339968801 +0000 UTC m=+1645.043134736" Nov 01 16:15:07.436977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.436783 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:07.758861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:07.758826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:08.117080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:08.117041 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:08.118877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:08.118852 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:15:08.118980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:08.118910 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 16:15:08.224937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:08.224897 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:15:08.225117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:08.224965 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 16:15:08.543826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:08.542654 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:08.847718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:08.847641 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:09.120259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:09.120224 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:15:09.120532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:09.120502 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 16:15:09.225158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:09.225120 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:15:09.225338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:09.225187 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 16:15:09.330679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:09.330583 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:09.331715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:09.331499 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:15:09.945928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:09.945890 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:10.121116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:10.120484 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:15:10.121116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:10.120570 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 16:15:10.128994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:10.128961 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:10.224834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:10.224701 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:15:10.224834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:10.224775 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 16:15:10.317936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:10.317892 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:10.424776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:10.415674 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:11.118902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:11.118136 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:15:11.118902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:11.118199 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 16:15:11.225538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:11.225485 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:15:11.225734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:11.225561 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 16:15:11.324790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:11.324630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:11.453786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:11.453677 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:11.651805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:11.651770 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:12.118401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:12.118210 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:15:12.118401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:12.118277 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 16:15:12.224968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:12.224927 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:15:12.225465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:12.224991 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 16:15:12.428202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:12.428118 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:13.117342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:13.117305 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body= Nov 01 16:15:13.117524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:13.117358 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" Nov 01 16:15:13.229984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:13.229945 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:15:13.230367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:13.230312 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:15:13.231183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:13.231160 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:15:13.280559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:13.280520 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:13.450907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:13.450832 4573 scope.go:122] "RemoveContainer" containerID="1d59a136c21236aa578f8637de56246c2f89aa320784231b08adbeba70eb9a20" Nov 01 16:15:13.975177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:13.975136 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:14.318215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:14.315512 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:14.318215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:14.317115 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:14.321293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:14.318365 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:15:14.461038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:14.461005 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" event={"ID":"3b6bb026-2955-4763-aa43-299714930a67","Type":"ContainerStarted","Data":"bb37bf646cf837ec7ebc5a04d184b60e074c21e44d869b0d10037a61a431a4a7"} Nov 01 16:15:14.480556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:14.480507 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" podStartSLOduration=14.480492664 podStartE2EDuration="14.480492664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:14.478271861 +0000 UTC m=+1652.181437798" watchObservedRunningTime="2026-11-01 16:15:14.480492664 +0000 UTC m=+1652.183658635" Nov 01 16:15:14.884921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:14.884857 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:15.473070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:15.473032 4573 generic.go:356] "Generic (PLEG): container finished" podID="3b6bb026-2955-4763-aa43-299714930a67" containerID="bb37bf646cf837ec7ebc5a04d184b60e074c21e44d869b0d10037a61a431a4a7" exitCode=60 Nov 01 16:15:15.473562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:15.473091 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" event={"ID":"3b6bb026-2955-4763-aa43-299714930a67","Type":"ContainerDied","Data":"bb37bf646cf837ec7ebc5a04d184b60e074c21e44d869b0d10037a61a431a4a7"} Nov 01 16:15:15.473562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:15.473164 4573 scope.go:122] "RemoveContainer" containerID="1d59a136c21236aa578f8637de56246c2f89aa320784231b08adbeba70eb9a20" Nov 01 16:15:15.473889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:15.473867 4573 scope.go:122] "RemoveContainer" containerID="bb37bf646cf837ec7ebc5a04d184b60e074c21e44d869b0d10037a61a431a4a7" Nov 01 16:15:15.474421 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:15.474391 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2_odh-ai-gateway-infra(3b6bb026-2955-4763-aa43-299714930a67)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" podUID="3b6bb026-2955-4763-aa43-299714930a67" Nov 01 16:15:15.493692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:15.493665 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2"] Nov 01 16:15:15.867511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:15.867482 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:16.664155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.664123 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:16.693880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.693850 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj"] Nov 01 16:15:16.694858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.694838 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:15:16.694913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.694860 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:15:16.694962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.694925 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:15:16.694962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.694932 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:15:16.695080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.695070 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:15:16.695113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.695085 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:15:16.695113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.695095 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:15:16.699246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.699223 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:16.704535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.704505 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj"] Nov 01 16:15:16.774902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.774819 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:16.794467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.794431 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6bb026-2955-4763-aa43-299714930a67-kube-api-access-wmvd2\" (UniqueName: \"kubernetes.io/projected/3b6bb026-2955-4763-aa43-299714930a67-kube-api-access-wmvd2\") pod \"3b6bb026-2955-4763-aa43-299714930a67\" (UID: \"3b6bb026-2955-4763-aa43-299714930a67\") " Nov 01 16:15:16.794866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.794837 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6bb026-2955-4763-aa43-299714930a67-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6bb026-2955-4763-aa43-299714930a67-maas-api-service-ca\") pod \"3b6bb026-2955-4763-aa43-299714930a67\" (UID: \"3b6bb026-2955-4763-aa43-299714930a67\") " Nov 01 16:15:16.795325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.795297 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16621e4-6f95-40c9-9fd8-a4e69ad56763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj\" (UID: \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:16.795434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.795374 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655w2\" (UniqueName: \"kubernetes.io/projected/f16621e4-6f95-40c9-9fd8-a4e69ad56763-kube-api-access-655w2\") pod \"maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj\" (UID: \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:16.796145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.796118 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6bb026-2955-4763-aa43-299714930a67-maas-api-service-ca" pod "3b6bb026-2955-4763-aa43-299714930a67" (UID: "3b6bb026-2955-4763-aa43-299714930a67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:16.796740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.796714 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6bb026-2955-4763-aa43-299714930a67-kube-api-access-wmvd2" pod "3b6bb026-2955-4763-aa43-299714930a67" (UID: "3b6bb026-2955-4763-aa43-299714930a67"). InnerVolumeSpecName "kube-api-access-wmvd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:16.897504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.897460 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16621e4-6f95-40c9-9fd8-a4e69ad56763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj\" (UID: \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:16.897785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.897625 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655w2\" (UniqueName: \"kubernetes.io/projected/f16621e4-6f95-40c9-9fd8-a4e69ad56763-kube-api-access-655w2\") pod \"maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj\" (UID: \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:16.898030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.897994 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmvd2\" (UniqueName: \"kubernetes.io/projected/3b6bb026-2955-4763-aa43-299714930a67-kube-api-access-wmvd2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:16.898030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.898025 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b6bb026-2955-4763-aa43-299714930a67-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:16.898386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.898353 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16621e4-6f95-40c9-9fd8-a4e69ad56763-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj\" (UID: \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:16.909052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:16.909031 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655w2\" (UniqueName: \"kubernetes.io/projected/f16621e4-6f95-40c9-9fd8-a4e69ad56763-kube-api-access-655w2\") pod \"maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj\" (UID: \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:17.021274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.021245 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:17.342120 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:15:17.342090 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16621e4_6f95_40c9_9fd8_a4e69ad56763.slice/crio-dc47f4ff29a31eafc85353151edeed2072716f9cae4a09d4de8a76dea5c24c19 WatchSource:0}: Error finding container dc47f4ff29a31eafc85353151edeed2072716f9cae4a09d4de8a76dea5c24c19: Status 404 returned error can't find the container with id dc47f4ff29a31eafc85353151edeed2072716f9cae4a09d4de8a76dea5c24c19 Nov 01 16:15:17.342948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.342921 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj"] Nov 01 16:15:17.450490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.450457 4573 scope.go:122] "RemoveContainer" containerID="f2455be3f40d66bab8b915a2b0da7420ffb7f5739674ce12c9d73197c01f8e84" Nov 01 16:15:17.498805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.498769 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" Nov 01 16:15:17.498996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.498802 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2" event={"ID":"3b6bb026-2955-4763-aa43-299714930a67","Type":"ContainerDied","Data":"082811b60bdae37f2fd85625f611454624606852717e81d2493be1643b22390c"} Nov 01 16:15:17.498996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.498869 4573 scope.go:122] "RemoveContainer" containerID="bb37bf646cf837ec7ebc5a04d184b60e074c21e44d869b0d10037a61a431a4a7" Nov 01 16:15:17.501594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.501569 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" event={"ID":"f16621e4-6f95-40c9-9fd8-a4e69ad56763","Type":"ContainerStarted","Data":"dc47f4ff29a31eafc85353151edeed2072716f9cae4a09d4de8a76dea5c24c19"} Nov 01 16:15:17.550301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.550270 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2"] Nov 01 16:15:17.555509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.555482 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-5e2d03-29892495-5mwg2"] Nov 01 16:15:17.696831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:17.696797 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:18.461838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:18.461799 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6bb026-2955-4763-aa43-299714930a67" path="/var/lib/kubelet/pods/3b6bb026-2955-4763-aa43-299714930a67/volumes" Nov 01 16:15:18.514775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:18.514725 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" event={"ID":"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6","Type":"ContainerStarted","Data":"9a398d3943f655dda054420abd980980758ae355105c6d1d4788c3bc9787d137"} Nov 01 16:15:18.517633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:18.517603 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" event={"ID":"f16621e4-6f95-40c9-9fd8-a4e69ad56763","Type":"ContainerStarted","Data":"ff2b4ac6d29ef1f4457fa01ec672c6246db99c449f523f84b7d1d8e8da033d71"} Nov 01 16:15:18.532074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:18.532016 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" podStartSLOduration=18.532002478 podStartE2EDuration="18.532002478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:18.531327726 +0000 UTC m=+1656.234493663" watchObservedRunningTime="2026-11-01 16:15:18.532002478 +0000 UTC m=+1656.235168413" Nov 01 16:15:18.549639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:18.549589 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" podStartSLOduration=18.549575233 podStartE2EDuration="18.549575233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:15:18.548414901 +0000 UTC m=+1656.251580862" watchObservedRunningTime="2026-11-01 16:15:18.549575233 +0000 UTC m=+1656.252741169" Nov 01 16:15:18.560024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:18.559992 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:18.793947 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:18.793921 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc8b9da_6b7b_4854_b6ac_6da05ec6a8b6.slice/crio-conmon-9a398d3943f655dda054420abd980980758ae355105c6d1d4788c3bc9787d137.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:15:19.375422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.375389 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:19.528942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.528907 4573 generic.go:356] "Generic (PLEG): container finished" podID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerID="9a398d3943f655dda054420abd980980758ae355105c6d1d4788c3bc9787d137" exitCode=60 Nov 01 16:15:19.529120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.528963 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" event={"ID":"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6","Type":"ContainerDied","Data":"9a398d3943f655dda054420abd980980758ae355105c6d1d4788c3bc9787d137"} Nov 01 16:15:19.529120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.529034 4573 scope.go:122] "RemoveContainer" containerID="f2455be3f40d66bab8b915a2b0da7420ffb7f5739674ce12c9d73197c01f8e84" Nov 01 16:15:19.529569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.529536 4573 scope.go:122] "RemoveContainer" containerID="9a398d3943f655dda054420abd980980758ae355105c6d1d4788c3bc9787d137" Nov 01 16:15:19.530157 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:19.530041 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn_odh-ai-gateway-infra(5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" Nov 01 16:15:19.532794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.532767 4573 generic.go:356] "Generic (PLEG): container finished" podID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerID="ff2b4ac6d29ef1f4457fa01ec672c6246db99c449f523f84b7d1d8e8da033d71" exitCode=60 Nov 01 16:15:19.532964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.532810 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" event={"ID":"f16621e4-6f95-40c9-9fd8-a4e69ad56763","Type":"ContainerDied","Data":"ff2b4ac6d29ef1f4457fa01ec672c6246db99c449f523f84b7d1d8e8da033d71"} Nov 01 16:15:19.533373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.533353 4573 scope.go:122] "RemoveContainer" containerID="ff2b4ac6d29ef1f4457fa01ec672c6246db99c449f523f84b7d1d8e8da033d71" Nov 01 16:15:19.548901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:19.548875 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn"] Nov 01 16:15:20.257767 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.257720 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:20.543549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.543481 4573 generic.go:356] "Generic (PLEG): container finished" podID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerID="515f1e0da0622017ad7a2cd4650e9a1a32d5caccf40f6b8c2e17cc00770e85e4" exitCode=60 Nov 01 16:15:20.543674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.543560 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" event={"ID":"f16621e4-6f95-40c9-9fd8-a4e69ad56763","Type":"ContainerDied","Data":"515f1e0da0622017ad7a2cd4650e9a1a32d5caccf40f6b8c2e17cc00770e85e4"} Nov 01 16:15:20.543674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.543615 4573 scope.go:122] "RemoveContainer" containerID="ff2b4ac6d29ef1f4457fa01ec672c6246db99c449f523f84b7d1d8e8da033d71" Nov 01 16:15:20.544037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.544021 4573 scope.go:122] "RemoveContainer" containerID="515f1e0da0622017ad7a2cd4650e9a1a32d5caccf40f6b8c2e17cc00770e85e4" Nov 01 16:15:20.544473 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:20.544448 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj_odh-ai-gateway-infra(f16621e4-6f95-40c9-9fd8-a4e69ad56763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" Nov 01 16:15:20.725822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.725794 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:20.811593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.811492 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-kube-api-access-nczrl\" (UniqueName: \"kubernetes.io/projected/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-kube-api-access-nczrl\") pod \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\" (UID: \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\") " Nov 01 16:15:20.811956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.811931 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-maas-api-service-ca\") pod \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\" (UID: \"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6\") " Nov 01 16:15:20.812337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.812310 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-maas-api-service-ca" pod "5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" (UID: "5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:20.812763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.812730 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:20.813853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.813830 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-kube-api-access-nczrl" pod "5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" (UID: "5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6"). InnerVolumeSpecName "kube-api-access-nczrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:20.914820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.914786 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nczrl\" (UniqueName: \"kubernetes.io/projected/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6-kube-api-access-nczrl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:20.968683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:20.968645 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:21.554584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:21.554548 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" Nov 01 16:15:21.555110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:21.554599 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn" event={"ID":"5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6","Type":"ContainerDied","Data":"056e689c9d4c506b9eaa5117bfc014d27825af0d406d55a3fd735e83d21aedd7"} Nov 01 16:15:21.555110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:21.554660 4573 scope.go:122] "RemoveContainer" containerID="9a398d3943f655dda054420abd980980758ae355105c6d1d4788c3bc9787d137" Nov 01 16:15:21.556778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:21.556742 4573 scope.go:122] "RemoveContainer" containerID="515f1e0da0622017ad7a2cd4650e9a1a32d5caccf40f6b8c2e17cc00770e85e4" Nov 01 16:15:21.557055 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:21.557037 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj_odh-ai-gateway-infra(f16621e4-6f95-40c9-9fd8-a4e69ad56763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" Nov 01 16:15:21.588417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:21.588393 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn"] Nov 01 16:15:21.594092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:21.594070 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c8d121-29892495-shxbn"] Nov 01 16:15:21.762452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:21.762418 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:22.677111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:22.677067 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" path="/var/lib/kubelet/pods/5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6/volumes" Nov 01 16:15:22.730647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:22.730601 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:23.427998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:23.427931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:23.781153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:23.781047 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:23.883706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:23.883676 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:24.156359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:24.156330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:24.365878 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:24.365843 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:15:24.920594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:24.920562 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:25.635484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:25.635449 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:26.029884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:26.029788 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2"] Nov 01 16:15:26.034623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:26.034594 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d98f52c9-v6nm2"] Nov 01 16:15:26.251001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:26.250969 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:26.461158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:26.461127 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74203b60-c4f9-4503-ac79-fcbccfb5a8f2" path="/var/lib/kubelet/pods/74203b60-c4f9-4503-ac79-fcbccfb5a8f2/volumes" Nov 01 16:15:26.488179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:26.488148 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:26.698601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:26.698572 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:27.234116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:27.234085 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:27.536835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:27.532545 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:27.862541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:27.862514 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:28.882914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:28.882882 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:29.772583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:29.772549 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:30.566003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:30.565967 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:31.365811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:31.365728 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:31.481230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:31.478495 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:31.993872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:31.993840 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:32.766823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:32.766733 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:33.588347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:33.588302 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:33.691846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:33.691796 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:33.770721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:33.770690 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:34.450992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:34.450965 4573 scope.go:122] "RemoveContainer" containerID="515f1e0da0622017ad7a2cd4650e9a1a32d5caccf40f6b8c2e17cc00770e85e4" Nov 01 16:15:34.485317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:34.485284 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:35.282606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:35.282572 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:35.837696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:35.837662 4573 generic.go:356] "Generic (PLEG): container finished" podID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerID="610957f540a049443ed20264d8a07b1861964bfbecda4be08fe163a59a52bf22" exitCode=60 Nov 01 16:15:35.837928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:35.837736 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" event={"ID":"f16621e4-6f95-40c9-9fd8-a4e69ad56763","Type":"ContainerDied","Data":"610957f540a049443ed20264d8a07b1861964bfbecda4be08fe163a59a52bf22"} Nov 01 16:15:35.837928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:35.837837 4573 scope.go:122] "RemoveContainer" containerID="515f1e0da0622017ad7a2cd4650e9a1a32d5caccf40f6b8c2e17cc00770e85e4" Nov 01 16:15:35.838290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:35.838272 4573 scope.go:122] "RemoveContainer" containerID="610957f540a049443ed20264d8a07b1861964bfbecda4be08fe163a59a52bf22" Nov 01 16:15:35.838683 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:35.838659 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj_odh-ai-gateway-infra(f16621e4-6f95-40c9-9fd8-a4e69ad56763)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" Nov 01 16:15:35.958177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:35.958147 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:36.770792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:36.770714 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:36.875145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:36.875060 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj"] Nov 01 16:15:37.072140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.072116 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:37.145151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.145064 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16621e4-6f95-40c9-9fd8-a4e69ad56763-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16621e4-6f95-40c9-9fd8-a4e69ad56763-maas-api-service-ca\") pod \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\" (UID: \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\") " Nov 01 16:15:37.145316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.145233 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16621e4-6f95-40c9-9fd8-a4e69ad56763-kube-api-access-655w2\" (UniqueName: \"kubernetes.io/projected/f16621e4-6f95-40c9-9fd8-a4e69ad56763-kube-api-access-655w2\") pod \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\" (UID: \"f16621e4-6f95-40c9-9fd8-a4e69ad56763\") " Nov 01 16:15:37.145683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.145652 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16621e4-6f95-40c9-9fd8-a4e69ad56763-maas-api-service-ca" pod "f16621e4-6f95-40c9-9fd8-a4e69ad56763" (UID: "f16621e4-6f95-40c9-9fd8-a4e69ad56763"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:15:37.146660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.146635 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f16621e4-6f95-40c9-9fd8-a4e69ad56763-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:37.148645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.148615 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16621e4-6f95-40c9-9fd8-a4e69ad56763-kube-api-access-655w2" pod "f16621e4-6f95-40c9-9fd8-a4e69ad56763" (UID: "f16621e4-6f95-40c9-9fd8-a4e69ad56763"). InnerVolumeSpecName "kube-api-access-655w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:15:37.248040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.248007 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655w2\" (UniqueName: \"kubernetes.io/projected/f16621e4-6f95-40c9-9fd8-a4e69ad56763-kube-api-access-655w2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:15:37.563456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.563376 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:37.882038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.882012 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" Nov 01 16:15:37.882503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.882047 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj" event={"ID":"f16621e4-6f95-40c9-9fd8-a4e69ad56763","Type":"ContainerDied","Data":"dc47f4ff29a31eafc85353151edeed2072716f9cae4a09d4de8a76dea5c24c19"} Nov 01 16:15:37.882503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.882107 4573 scope.go:122] "RemoveContainer" containerID="610957f540a049443ed20264d8a07b1861964bfbecda4be08fe163a59a52bf22" Nov 01 16:15:37.908128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.908097 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj"] Nov 01 16:15:37.912530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:37.912504 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-99afca-29892495-x5kvj"] Nov 01 16:15:38.247686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:38.247603 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:38.460298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:38.460263 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" path="/var/lib/kubelet/pods/f16621e4-6f95-40c9-9fd8-a4e69ad56763/volumes" Nov 01 16:15:38.945772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:38.942217 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:39.745467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:39.745401 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:40.464408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:40.464372 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:41.085007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:41.084968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:41.940816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:41.924305 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:42.687902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:42.687868 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:43.354571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:43.354531 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:44.052450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:44.052419 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:44.685971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:44.685928 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:45.360308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:45.360275 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:46.006191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:46.006156 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:46.684469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:46.684421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:47.384707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:47.384675 4573 scope.go:122] "RemoveContainer" containerID="936e23f79abd6dea8b1e5da10e4d0742c48e2d3d84226452dab3a6f12fff9d07" Nov 01 16:15:47.398695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:47.398673 4573 scope.go:122] "RemoveContainer" containerID="9e2f7eee0515940777205d4100d83e2c96694af789b15da0219e21f11583a9eb" Nov 01 16:15:47.409912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:47.409837 4573 scope.go:122] "RemoveContainer" containerID="8276c377ff3e9808567f56efa4c2c9533bbcf70969d2ee7e77514f2a43e72515" Nov 01 16:15:47.416554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:47.416528 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:48.157838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:48.157806 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:49.059491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:49.059461 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:49.762597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:49.762562 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:50.421545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:50.421513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:51.112693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:51.112643 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:51.761887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:51.761846 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:52.502475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:52.502442 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:53.347343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:53.347314 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:54.076989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:54.076957 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:54.367218 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:15:54.367171 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:15:54.880288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:54.880258 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:55.794581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:55.794547 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:56.512959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:56.512927 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:57.172600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:57.172565 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:57.868145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:57.868105 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:58.545700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:58.545664 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:59.192007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:59.191972 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:15:59.906017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:15:59.905978 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:00.672589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:00.672552 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:00.794775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:00.794259 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:01.576499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:01.576465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:01.922977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:01.922938 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:02.689598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:02.689569 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:03.347497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:03.347465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:04.035060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:04.035029 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:04.763741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:04.763710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:05.469112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:05.469082 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:06.182236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:06.182202 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:06.962983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:06.962949 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:07.678651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:07.678618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:08.546776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:08.544420 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:09.181381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:09.181347 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:10.009517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:10.009482 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:10.716371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:10.716325 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:11.475793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:11.475736 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:12.151542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:12.151508 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:12.856545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:12.856513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:13.570604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:13.570565 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:14.270895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:14.270862 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:15.121771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:15.118263 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:15.664017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:15.663987 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:16.322884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:16.322843 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:16.855446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:16.855411 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:17.670594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:17.670561 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:18.368282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:18.368253 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:19.153454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:19.153418 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:19.955833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:19.955798 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:20.588621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:20.588579 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:21.456196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:21.456158 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:22.025464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:22.025428 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:22.833413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:22.833375 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:23.537982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:23.537942 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:24.186160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:24.186121 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:24.373010 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:16:24.372971 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:16:24.985129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:24.985100 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:25.756903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:25.756867 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:26.560070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:26.560039 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:27.326082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:27.326050 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:27.973616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:27.973581 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:28.709693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:28.709663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:29.457656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:29.457621 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:30.144962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:30.144927 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:30.839294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:30.839258 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:31.535852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:31.535821 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:31.642376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:31.642341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:32.173320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:32.173289 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:32.880922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:32.880887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:33.672969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:33.672930 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:34.278151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:34.278117 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:16:34.310799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:34.310733 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:35.241592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:35.241545 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:36.174902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:36.174870 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:36.991268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:36.991180 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:37.886972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:37.886931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:38.688394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:38.688360 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:39.487959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:39.487922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:40.258690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:40.258654 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:41.149525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:41.149490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:42.050357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:42.050324 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:42.793869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:42.793832 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:42.909209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:42.909176 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:43.061838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:43.054723 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9"] Nov 01 16:16:43.092249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:43.092180 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0dd02c1-rf9b9"] Nov 01 16:16:43.907879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:43.907847 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:44.076687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:44.076626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:44.190670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:44.190573 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:44.614016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:44.603694 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13fbf8b-7440-4816-9ea4-49539c2b8bf0" path="/var/lib/kubelet/pods/c13fbf8b-7440-4816-9ea4-49539c2b8bf0/volumes" Nov 01 16:16:44.958685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:44.958588 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:45.660742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:45.660703 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:46.462547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:46.462517 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:47.154360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:47.154329 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:47.491373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:47.491285 4573 scope.go:122] "RemoveContainer" containerID="93d326e82425694ffe090a3c02a71c8dfe755cc270b9f80ee8593e6dadbf7032" Nov 01 16:16:47.505987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:47.505946 4573 scope.go:122] "RemoveContainer" containerID="c14c4ff006995e1b0a3a22fb352197f5fe758710e9bd56ca38622d23cbaa50b3" Nov 01 16:16:47.519349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:47.519324 4573 scope.go:122] "RemoveContainer" containerID="f03c5a81374fe9d7bc220a95baf1da06a2064a9134c3c712dce204b38999a949" Nov 01 16:16:48.082123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:48.082054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:48.816129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:48.816087 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:48.973418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:48.973384 4573 generic.go:356] "Generic (PLEG): container finished" podID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerID="dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046" exitCode=0 Nov 01 16:16:48.973625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:48.973450 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" event={"ID":"1febdd5f-7998-4501-a41d-d87ef58a3703","Type":"ContainerDied","Data":"dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046"} Nov 01 16:16:48.974039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:48.974020 4573 scope.go:122] "RemoveContainer" containerID="dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046" Nov 01 16:16:48.979221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:48.979192 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:48.979323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:48.979266 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:48.979323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:48.979298 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:16:49.468565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:49.468529 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:50.000815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:50.000781 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" event={"ID":"1febdd5f-7998-4501-a41d-d87ef58a3703","Type":"ContainerStarted","Data":"454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10"} Nov 01 16:16:50.211925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:50.211887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:50.436198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:50.436171 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:16:50.437490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:50.437457 4573 patch_prober.go:29] interesting pod/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 16:16:50.437614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:50.437504 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 16:16:50.884784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:50.881234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:51.436658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:51.436625 4573 patch_prober.go:29] interesting pod/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 16:16:51.437086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:51.436682 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 16:16:51.583694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:51.583659 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:52.221974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:52.221936 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:52.436588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:52.436553 4573 patch_prober.go:29] interesting pod/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 16:16:52.436808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:52.436610 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 16:16:52.986789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:52.985574 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:53.437297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:53.437267 4573 patch_prober.go:29] interesting pod/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 16:16:53.437688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:53.437317 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 16:16:53.630580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:53.630522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:54.285246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:54.285209 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:54.339773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:54.334070 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:16:54.441808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:54.441214 4573 patch_prober.go:29] interesting pod/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body= Nov 01 16:16:54.441808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:54.441278 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" Nov 01 16:16:54.498860 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:16:54.498824 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:16:55.081359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:55.081320 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:55.442367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:55.442339 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:16:55.443310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:55.443290 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:16:55.796615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:55.796525 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:56.580109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:56.580059 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:57.263466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:57.263432 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:57.952141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:57.952109 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:58.662241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:58.662206 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:16:59.351403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:16:59.351368 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:00.039074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:00.038974 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:00.787986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:00.787949 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:01.559187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:01.559152 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:02.486267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:02.486229 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:02.596513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:02.596478 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:02.748770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:02.748674 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:03.465130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:03.465040 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:04.204003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:04.203966 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:04.314591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:04.314540 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:04.525062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:04.524985 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:05.216215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:05.216182 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:05.981506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:05.981475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:06.660818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:06.660787 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:07.453639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:07.453603 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:08.274983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:08.274952 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:08.946963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:08.946926 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:09.594064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:09.594030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:10.467160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:10.467130 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:10.789160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:10.789082 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:11.572872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:11.572831 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:12.258181 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:12.258146 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:13.036645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:13.036607 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:13.945770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:13.945669 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:14.271559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.271476 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6"] Nov 01 16:17:14.271721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.271683 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" podUID="3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" containerName="discovery" containerID="cri-o://8f7d405a247bc04c5c86ad7b9a23cd916c1aeb8ae078e4ff1a84cd53df32d4c8" gracePeriod=30 Nov 01 16:17:14.417180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.417151 4573 generic.go:356] "Generic (PLEG): container finished" podID="3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" containerID="8f7d405a247bc04c5c86ad7b9a23cd916c1aeb8ae078e4ff1a84cd53df32d4c8" exitCode=0 Nov 01 16:17:14.417314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.417190 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" event={"ID":"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2","Type":"ContainerDied","Data":"8f7d405a247bc04c5c86ad7b9a23cd916c1aeb8ae078e4ff1a84cd53df32d4c8"} Nov 01 16:17:14.584742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.584717 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:17:14.622068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.622031 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9"] Nov 01 16:17:14.623234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623206 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.623234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623234 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.623401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623252 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.623401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623259 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.623401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623279 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" containerName="discovery" Nov 01 16:17:14.623401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623287 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" containerName="discovery" Nov 01 16:17:14.623401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623298 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.623401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623305 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.623401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623314 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:17:14.623401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623320 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6bb026-2955-4763-aa43-299714930a67" containerName="cleanup" Nov 01 16:17:14.623707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623410 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.623707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623417 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.623707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623590 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.623707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623606 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.623707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623622 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.623707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623632 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.623707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.623641 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" containerName="discovery" Nov 01 16:17:14.628069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.628041 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.629200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.629177 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5"] Nov 01 16:17:14.630335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.630309 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.630335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.630330 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.630490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.630347 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.630490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.630354 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.630607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.630575 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16621e4-6f95-40c9-9fd8-a4e69ad56763" containerName="cleanup" Nov 01 16:17:14.630607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.630595 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc8b9da-6b7b-4854-b6ac-6da05ec6a8b6" containerName="cleanup" Nov 01 16:17:14.635009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.634988 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.636321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.636298 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2"] Nov 01 16:17:14.639250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.639230 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-99e837-openshift-default-dockercfg-dmd8v\"" Nov 01 16:17:14.642071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.642052 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:14.643568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.643542 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8"] Nov 01 16:17:14.649872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.649855 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.656525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.656497 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ae79df\"" Nov 01 16:17:14.661507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.661484 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5"] Nov 01 16:17:14.686665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.686632 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2"] Nov 01 16:17:14.770444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.745673 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ae79df-dockercfg-tkqc9\"" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.750957 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-ae79df-serving-cert\"" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.751081 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a4240a-openshift-default-dockercfg-r7gj2\"" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.772323 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-kubeconfig\") pod \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.819343 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-kube-api-access-rd8xl\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-kube-api-access-rd8xl\") pod \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.819454 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-local-certs\") pod \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.819490 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-token\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-token\") pod \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.819607 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-dns-cert\") pod \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.819670 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-ca-configmap\") pod \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.819724 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-cacerts\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-cacerts\") pod \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\" (UID: \"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2\") " Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820021 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgrf\" (UniqueName: \"kubernetes.io/projected/45395229-36ee-4bce-9004-e3be3041e165-kube-api-access-mxgrf\") pod \"payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2\" (UID: \"45395229-36ee-4bce-9004-e3be3041e165\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820051 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572793d5-f154-417a-850e-12e176c14413-istiod-ca-cert\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820075 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rmrc\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-kube-api-access-7rmrc\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820107 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-certs\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820149 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45395229-36ee-4bce-9004-e3be3041e165-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2\" (UID: \"45395229-36ee-4bce-9004-e3be3041e165\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820171 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-certs\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820192 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820210 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-data\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820242 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-metrics-certs\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820261 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-token\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820370 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gz5k\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-kube-api-access-9gz5k\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820402 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dccl\" (UniqueName: \"kubernetes.io/projected/b37c04ce-7216-422c-b973-974e69057427-kube-api-access-5dccl\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820462 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-envoy\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820517 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-podinfo\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820552 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-data\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820601 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-maas-api-tls\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820624 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-credential-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820649 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572793d5-f154-417a-850e-12e176c14413-istio-podinfo\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820682 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-envoy\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820697 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37c04ce-7216-422c-b973-974e69057427-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820746 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-istio-token\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.820827 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-credential-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.822234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.821615 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-ca-configmap" pod "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" (UID: "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:17:14.824654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.777221 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:14.824654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.815737 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9"] Nov 01 16:17:14.836394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.829392 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-dns-cert" pod "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" (UID: "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:17:14.843943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.840181 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.843943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.840346 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istiod-ca-cert\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.843943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.840587 4573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-dns-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:14.843943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.840606 4573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-csr-ca-configmap\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:14.844369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.844274 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-token" pod "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" (UID: "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:17:14.846864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.846814 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-local-certs" pod "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" (UID: "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:17:14.850619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.850577 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-kube-api-access-rd8xl" pod "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" (UID: "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2"). InnerVolumeSpecName "kube-api-access-rd8xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:17:14.852197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.852173 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-cacerts" pod "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" (UID: "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:17:14.854459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.854428 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8"] Nov 01 16:17:14.858011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.857984 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-kubeconfig" pod "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" (UID: "3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:17:14.942367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942332 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-data\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.942529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942381 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-metrics-certs\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.942529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942498 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-token\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.942673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942654 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gz5k\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-kube-api-access-9gz5k\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.942816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942709 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dccl\" (UniqueName: \"kubernetes.io/projected/b37c04ce-7216-422c-b973-974e69057427-kube-api-access-5dccl\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.942816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942806 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-envoy\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.942933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942813 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-data\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.942933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942894 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-podinfo\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.943070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.942945 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-data\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.943070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943015 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-maas-api-tls\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.943184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943123 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-credential-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.943184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943170 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572793d5-f154-417a-850e-12e176c14413-istio-podinfo\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.943302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943216 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-envoy\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.943302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943245 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37c04ce-7216-422c-b973-974e69057427-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943316 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-istio-token\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943385 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-credential-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943405 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-credential-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943448 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943494 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istiod-ca-cert\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943539 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgrf\" (UniqueName: \"kubernetes.io/projected/45395229-36ee-4bce-9004-e3be3041e165-kube-api-access-mxgrf\") pod \"payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2\" (UID: \"45395229-36ee-4bce-9004-e3be3041e165\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943542 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-data\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943570 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572793d5-f154-417a-850e-12e176c14413-istiod-ca-cert\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943605 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rmrc\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-kube-api-access-7rmrc\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943648 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-certs\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.943773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943713 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45395229-36ee-4bce-9004-e3be3041e165-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2\" (UID: \"45395229-36ee-4bce-9004-e3be3041e165\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:14.944442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943744 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-certs\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.944442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943834 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.944442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.943868 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-credential-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.944442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.944270 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-socket\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.944809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.944714 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istiod-ca-cert\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.944809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.944788 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-certs\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.945403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.945379 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37c04ce-7216-422c-b973-974e69057427-openshift-service-ca\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.945916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.945885 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-socket\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.946349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.945913 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45395229-36ee-4bce-9004-e3be3041e165-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2\" (UID: \"45395229-36ee-4bce-9004-e3be3041e165\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:14.946349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.945923 4573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-local-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:14.946349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.946108 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:14.946349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.946135 4573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-cacerts\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:14.946349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.946151 4573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-istio-kubeconfig\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:14.946349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.946189 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8xl\" (UniqueName: \"kubernetes.io/projected/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2-kube-api-access-rd8xl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:14.946349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.946190 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-certs\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.946349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.946244 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572793d5-f154-417a-850e-12e176c14413-istiod-ca-cert\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.946852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.946726 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-envoy\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.947137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.947091 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-podinfo\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.947693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.947671 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-maas-api-tls\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.947840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.947713 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-metrics-certs\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.947983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.947958 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572793d5-f154-417a-850e-12e176c14413-istio-podinfo\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.948300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.948283 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-envoy\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.951934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.951911 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gz5k\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-kube-api-access-9gz5k\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.952214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.952191 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-token\") pod \"e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:14.952410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.952389 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dccl\" (UniqueName: \"kubernetes.io/projected/b37c04ce-7216-422c-b973-974e69057427-kube-api-access-5dccl\") pod \"maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.952947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.952921 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgrf\" (UniqueName: \"kubernetes.io/projected/45395229-36ee-4bce-9004-e3be3041e165-kube-api-access-mxgrf\") pod \"payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2\" (UID: \"45395229-36ee-4bce-9004-e3be3041e165\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:14.954071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.954054 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-istio-token\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.954322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.954305 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rmrc\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-kube-api-access-7rmrc\") pod \"e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:14.958209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.958193 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:14.972656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.972635 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:14.983363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:14.983339 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:15.248267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.248234 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:15.258988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.258962 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5"] Nov 01 16:17:15.264307 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:17:15.264272 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb37c04ce_7216_422c_b973_974e69057427.slice/crio-8dc7917858ee4f865bc75194704d2c41dd8af16d8b8625f856e847c1b65249b3 WatchSource:0}: Error finding container 8dc7917858ee4f865bc75194704d2c41dd8af16d8b8625f856e847c1b65249b3: Status 404 returned error can't find the container with id 8dc7917858ee4f865bc75194704d2c41dd8af16d8b8625f856e847c1b65249b3 Nov 01 16:17:15.279472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.279441 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2"] Nov 01 16:17:15.279967 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:17:15.279941 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45395229_36ee_4bce_9004_e3be3041e165.slice/crio-b2d2d73908493b5f0d0370acd6d50690430ab56dcf6923ccdf6a732e6d1ef350 WatchSource:0}: Error finding container b2d2d73908493b5f0d0370acd6d50690430ab56dcf6923ccdf6a732e6d1ef350: Status 404 returned error can't find the container with id b2d2d73908493b5f0d0370acd6d50690430ab56dcf6923ccdf6a732e6d1ef350 Nov 01 16:17:15.311911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.311884 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8"] Nov 01 16:17:15.314133 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:17:15.314102 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572793d5_f154_417a_850e_12e176c14413.slice/crio-800f97bd193d1badc1a60ba43f92fcf07f171a6ae162a9f6efe702e4ad195e4d WatchSource:0}: Error finding container 800f97bd193d1badc1a60ba43f92fcf07f171a6ae162a9f6efe702e4ad195e4d: Status 404 returned error can't find the container with id 800f97bd193d1badc1a60ba43f92fcf07f171a6ae162a9f6efe702e4ad195e4d Nov 01 16:17:15.318296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.318250 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:17:15.318565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.318365 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:17:15.318565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.318413 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:17:15.452662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.452627 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" event={"ID":"3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2","Type":"ContainerDied","Data":"4b50589f3002dd76127f8a0be706e1379d596578779887a4370e43289b27d46f"} Nov 01 16:17:15.452799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.452686 4573 scope.go:122] "RemoveContainer" containerID="8f7d405a247bc04c5c86ad7b9a23cd916c1aeb8ae078e4ff1a84cd53df32d4c8" Nov 01 16:17:15.452878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.452834 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6" Nov 01 16:17:15.474524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.474453 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" event={"ID":"b37c04ce-7216-422c-b973-974e69057427","Type":"ContainerStarted","Data":"8dc7917858ee4f865bc75194704d2c41dd8af16d8b8625f856e847c1b65249b3"} Nov 01 16:17:15.477708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.477679 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" event={"ID":"572793d5-f154-417a-850e-12e176c14413","Type":"ContainerStarted","Data":"800f97bd193d1badc1a60ba43f92fcf07f171a6ae162a9f6efe702e4ad195e4d"} Nov 01 16:17:15.497042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.496984 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6"] Nov 01 16:17:15.502618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.502510 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" event={"ID":"45395229-36ee-4bce-9004-e3be3041e165","Type":"ContainerStarted","Data":"b2d2d73908493b5f0d0370acd6d50690430ab56dcf6923ccdf6a732e6d1ef350"} Nov 01 16:17:15.502618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.502580 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:15.506792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.506767 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fv7j6"] Nov 01 16:17:15.523898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.523851 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" podStartSLOduration=31.523834698 podStartE2EDuration="31.523834698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:17:15.521096524 +0000 UTC m=+1773.224262460" watchObservedRunningTime="2026-11-01 16:17:15.523834698 +0000 UTC m=+1773.227000636" Nov 01 16:17:15.561740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.561709 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9"] Nov 01 16:17:15.562373 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:17:15.562346 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dfdff89_3026_4c1e_b2f1_3d77ace7963b.slice/crio-93d86b265d7d7f19593c2f486cfaf0e7ba6f39c7c1675c869cdbf8ef2db53ec7 WatchSource:0}: Error finding container 93d86b265d7d7f19593c2f486cfaf0e7ba6f39c7c1675c869cdbf8ef2db53ec7: Status 404 returned error can't find the container with id 93d86b265d7d7f19593c2f486cfaf0e7ba6f39c7c1675c869cdbf8ef2db53ec7 Nov 01 16:17:15.566106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.565997 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:17:15.566106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.566087 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:17:15.566299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.566121 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:17:15.624580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:15.624555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:16.381944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.381883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:16.650310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.650219 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2" path="/var/lib/kubelet/pods/3d6c2dce-ecb9-4ca8-96a4-c2e0a80d6ca2/volumes" Nov 01 16:17:16.653243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.653212 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" event={"ID":"0dfdff89-3026-4c1e-b2f1-3d77ace7963b","Type":"ContainerStarted","Data":"23fd6971e35bb8db474a03f323737e27ddb38084dcbeebeee4f66ebeac7f83cb"} Nov 01 16:17:16.653332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.653278 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" event={"ID":"0dfdff89-3026-4c1e-b2f1-3d77ace7963b","Type":"ContainerStarted","Data":"93d86b265d7d7f19593c2f486cfaf0e7ba6f39c7c1675c869cdbf8ef2db53ec7"} Nov 01 16:17:16.655734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.655708 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" event={"ID":"45395229-36ee-4bce-9004-e3be3041e165","Type":"ContainerStarted","Data":"7c98164b144899f8984c5c44869c6d88b63ba685b7a28cf6cc435349d3360dfa"} Nov 01 16:17:16.658952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.658928 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" event={"ID":"b37c04ce-7216-422c-b973-974e69057427","Type":"ContainerStarted","Data":"05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c"} Nov 01 16:17:16.659679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.659539 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:16.661003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.660981 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" event={"ID":"572793d5-f154-417a-850e-12e176c14413","Type":"ContainerStarted","Data":"951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9"} Nov 01 16:17:16.722737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.722669 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" podStartSLOduration=119.722647386 podStartE2EDuration="1m59.722647386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:17:16.676515608 +0000 UTC m=+1774.379681567" watchObservedRunningTime="2026-11-01 16:17:16.722647386 +0000 UTC m=+1774.425813323" Nov 01 16:17:16.729578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.729530 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" podStartSLOduration=24.729515626 podStartE2EDuration="24.729515626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:17:16.705597243 +0000 UTC m=+1774.408763182" watchObservedRunningTime="2026-11-01 16:17:16.729515626 +0000 UTC m=+1774.432681611" Nov 01 16:17:16.736022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:16.735971 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" podStartSLOduration=12.182515173 podStartE2EDuration="12.735957791s" totalImagesPullingTime="553.442618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:17:16.732055982 +0000 UTC m=+1774.435221912" watchObservedRunningTime="2026-11-01 16:17:16.735957791 +0000 UTC m=+1774.439123726" Nov 01 16:17:17.005220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.005124 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:17.018259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.018213 4573 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:17:17.018557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.018529 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 01 16:17:17.161706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.161668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:17.256237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.256133 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:17.265788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.263043 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 01 16:17:17.265788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.263119 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 01 16:17:17.691774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.690694 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:17.790777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.783044 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:17.954161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.954070 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:17.991788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.986125 4573 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:17:17.991788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:17.986189 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 01 16:17:18.249250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:18.249164 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 01 16:17:18.249250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:18.249215 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 01 16:17:18.654813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:18.654782 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:19.001836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:19.001193 4573 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:17:19.001836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:19.001255 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 01 16:17:19.249075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:19.249036 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 01 16:17:19.249268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:19.249087 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 01 16:17:19.444024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:19.443986 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:19.997775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:19.997360 4573 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:17:19.997775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:19.997433 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 01 16:17:20.152064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:20.152034 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:20.277835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:20.277063 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 01 16:17:20.277835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:20.277126 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 01 16:17:20.888775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:20.888581 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:20.988778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:20.986190 4573 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:17:20.988778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:20.986253 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 01 16:17:21.249940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:21.249852 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 01 16:17:21.249940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:21.249904 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 01 16:17:21.683775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:21.667209 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:21.984704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:21.984614 4573 patch_prober.go:29] interesting pod/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Nov 01 16:17:21.984704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:21.984680 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Nov 01 16:17:22.248698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:22.248594 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Nov 01 16:17:22.248698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:22.248661 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Nov 01 16:17:22.355277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:22.355237 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:23.004738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.004708 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:23.006040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.006013 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:23.006899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.006878 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:17:23.079213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.079173 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:23.258780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.258681 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:23.261161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.261137 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:23.264673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.264652 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:17:23.754785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.754709 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:23.761260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:23.761231 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:17:24.365403 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:17:24.365365 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:17:24.458436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:24.458397 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:25.179115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:25.179064 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:25.851364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:25.851333 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:26.494385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:26.494352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:27.063422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:27.063384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:27.163789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:27.163729 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:27.271370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:27.271335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:27.715897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:27.715865 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:17:28.285774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:28.282868 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:28.573520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:28.573436 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:28.984440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:28.984408 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:29.779349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:29.779310 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:30.359029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:30.358996 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:30.494676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:30.494646 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:30.594920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:30.594888 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:31.354490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:31.354457 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:31.603732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:31.603700 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:32.186841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:32.186807 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:33.152005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:33.151971 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:33.854163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:33.854133 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:34.765871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:34.765841 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:35.459991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:35.459958 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:36.192009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:36.191977 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:37.053861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:37.053830 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:38.309821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:38.309788 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:38.486350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:38.486320 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:39.177421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:39.177387 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:39.850656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:39.850622 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:40.657704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:40.657671 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:41.374655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:41.374612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:42.159921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:42.159860 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:42.876186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:42.876130 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:43.174064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:43.173969 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:17:43.182144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:43.182116 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:17:43.186363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:43.186342 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:17:43.194647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:43.194623 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:17:43.654554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:43.654523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:44.345367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:44.345334 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:45.151417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:45.151381 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:45.493043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:45.492956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:46.086531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:46.086499 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:46.838645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:46.838612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:47.167908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:47.167879 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:17:47.609922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:47.609015 4573 scope.go:122] "RemoveContainer" containerID="e74a89009a90ebbeccf9fe183e2b97d55d3bee5eb8b8b633c15c0391897800ec" Nov 01 16:17:47.642654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:47.642625 4573 scope.go:122] "RemoveContainer" containerID="2134c61c942a7782d1064b4655db0a30ce25350eb41668048ebda790ed58f8ab" Nov 01 16:17:47.655072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:47.654939 4573 scope.go:122] "RemoveContainer" containerID="6eb3837d62cdcd7249b46d265cc4233003b7b1153950e69c6326d1982197f761" Nov 01 16:17:47.665303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:47.665276 4573 scope.go:122] "RemoveContainer" containerID="238b2271370eb2e5702912b6fab09d60306172acdb2a58ca94b222cb74e85ac3" Nov 01 16:17:47.849926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:47.849890 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:48.653167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:48.653137 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:49.478541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:49.478512 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:50.275952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:50.275921 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:51.068277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:51.068242 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:51.768984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:51.768955 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:52.529708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:52.529678 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:53.435870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:53.435838 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:54.071875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:54.071845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:54.366320 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:17:54.366284 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:17:54.769898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:54.769817 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:55.123647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.123614 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb"] Nov 01 16:17:55.130670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.130647 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.134822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.134798 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:17:55.134976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.134898 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:17:55.136456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.136430 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb"] Nov 01 16:17:55.201387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.201351 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.201554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.201514 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.201819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.201798 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7cpw\" (UniqueName: \"kubernetes.io/projected/126681f3-42a3-4a99-bfba-cc2763e3970f-kube-api-access-l7cpw\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.303459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.303426 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.303619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.303511 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.303689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.303676 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7cpw\" (UniqueName: \"kubernetes.io/projected/126681f3-42a3-4a99-bfba-cc2763e3970f-kube-api-access-l7cpw\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.304204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.304179 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.305965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.305937 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.312326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.312301 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7cpw\" (UniqueName: \"kubernetes.io/projected/126681f3-42a3-4a99-bfba-cc2763e3970f-kube-api-access-l7cpw\") pod \"maas-api-revoke-sub-25f05218-tfwwb\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.450508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.450427 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:55.663439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.663403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:55.801798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:55.801766 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb"] Nov 01 16:17:55.802029 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:17:55.801895 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod126681f3_42a3_4a99_bfba_cc2763e3970f.slice/crio-54d0e2e2fb4f0c5e1d46a0f67b82d2c42d2ae8f5ae72628752bf7d5e7bf0f9b2 WatchSource:0}: Error finding container 54d0e2e2fb4f0c5e1d46a0f67b82d2c42d2ae8f5ae72628752bf7d5e7bf0f9b2: Status 404 returned error can't find the container with id 54d0e2e2fb4f0c5e1d46a0f67b82d2c42d2ae8f5ae72628752bf7d5e7bf0f9b2 Nov 01 16:17:56.335960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:56.335920 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" event={"ID":"126681f3-42a3-4a99-bfba-cc2763e3970f","Type":"ContainerStarted","Data":"54d0e2e2fb4f0c5e1d46a0f67b82d2c42d2ae8f5ae72628752bf7d5e7bf0f9b2"} Nov 01 16:17:56.458414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:56.458338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:57.140454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:57.140378 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:57.347667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:57.347629 4573 generic.go:356] "Generic (PLEG): container finished" podID="126681f3-42a3-4a99-bfba-cc2763e3970f" containerID="d1e6bfee60d4f2f280913d910a33561f3f5fff9b0c2bd33de371835dfd63cc7d" exitCode=0 Nov 01 16:17:57.348049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:57.347760 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" event={"ID":"126681f3-42a3-4a99-bfba-cc2763e3970f","Type":"ContainerDied","Data":"d1e6bfee60d4f2f280913d910a33561f3f5fff9b0c2bd33de371835dfd63cc7d"} Nov 01 16:17:57.847007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:57.846973 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:58.474117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.474077 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:58.609764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.607452 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:58.707249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.707206 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-xks-ca\") pod \"126681f3-42a3-4a99-bfba-cc2763e3970f\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " Nov 01 16:17:58.707451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.707404 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-openshift-ca\") pod \"126681f3-42a3-4a99-bfba-cc2763e3970f\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " Nov 01 16:17:58.707672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.707648 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/126681f3-42a3-4a99-bfba-cc2763e3970f-kube-api-access-l7cpw\" (UniqueName: \"kubernetes.io/projected/126681f3-42a3-4a99-bfba-cc2763e3970f-kube-api-access-l7cpw\") pod \"126681f3-42a3-4a99-bfba-cc2763e3970f\" (UID: \"126681f3-42a3-4a99-bfba-cc2763e3970f\") " Nov 01 16:17:58.707814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.707774 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-openshift-ca" pod "126681f3-42a3-4a99-bfba-cc2763e3970f" (UID: "126681f3-42a3-4a99-bfba-cc2763e3970f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:17:58.708453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.708425 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:58.709449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.709421 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-xks-ca" pod "126681f3-42a3-4a99-bfba-cc2763e3970f" (UID: "126681f3-42a3-4a99-bfba-cc2763e3970f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:17:58.709854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.709827 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/126681f3-42a3-4a99-bfba-cc2763e3970f-kube-api-access-l7cpw" pod "126681f3-42a3-4a99-bfba-cc2763e3970f" (UID: "126681f3-42a3-4a99-bfba-cc2763e3970f"). InnerVolumeSpecName "kube-api-access-l7cpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:17:58.810234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.810202 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7cpw\" (UniqueName: \"kubernetes.io/projected/126681f3-42a3-4a99-bfba-cc2763e3970f-kube-api-access-l7cpw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:58.810234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:58.810232 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/126681f3-42a3-4a99-bfba-cc2763e3970f-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:17:59.186147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:59.186107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:17:59.366701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:59.366671 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" Nov 01 16:17:59.366949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:59.366704 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb" event={"ID":"126681f3-42a3-4a99-bfba-cc2763e3970f","Type":"ContainerDied","Data":"54d0e2e2fb4f0c5e1d46a0f67b82d2c42d2ae8f5ae72628752bf7d5e7bf0f9b2"} Nov 01 16:17:59.366949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:59.366770 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d0e2e2fb4f0c5e1d46a0f67b82d2c42d2ae8f5ae72628752bf7d5e7bf0f9b2" Nov 01 16:17:59.951423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:17:59.951391 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:00.795171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:00.795141 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:01.452317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:01.452277 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:02.153939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:02.153901 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:02.880478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:02.880439 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:03.563617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:03.563583 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:04.236659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:04.236629 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:04.962740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:04.962712 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:05.694793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:05.694718 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:05.914316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:05.914187 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:06.242314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.242279 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px"] Nov 01 16:18:06.243326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.243303 4573 cpu_manager.go:414] "containerMap: removing container" podUID="126681f3-42a3-4a99-bfba-cc2763e3970f" containerName="revoke-keys" Nov 01 16:18:06.243326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.243324 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="126681f3-42a3-4a99-bfba-cc2763e3970f" containerName="revoke-keys" Nov 01 16:18:06.243535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.243467 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="126681f3-42a3-4a99-bfba-cc2763e3970f" containerName="revoke-keys" Nov 01 16:18:06.247485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.247465 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.251806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.251778 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:18:06.252034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.252017 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:18:06.265429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.264302 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px"] Nov 01 16:18:06.391231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.391195 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.391403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.391275 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdhxx\" (UniqueName: \"kubernetes.io/projected/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-kube-api-access-gdhxx\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.391525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.391501 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.492809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.492727 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.492985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.492969 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.493039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.493027 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdhxx\" (UniqueName: \"kubernetes.io/projected/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-kube-api-access-gdhxx\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.493603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.493576 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.495159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.495138 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.502096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.502076 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdhxx\" (UniqueName: \"kubernetes.io/projected/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-kube-api-access-gdhxx\") pod \"maas-api-revoke-sub-cfcf586d-z77px\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.584116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.584077 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:06.800046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.799960 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:06.934758 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:06.934715 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px"] Nov 01 16:18:06.935988 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:18:06.935960 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c5e879_c69f_4c29_b91e_aa7d7cbbb06a.slice/crio-ab345ad15e651ff3a4a74c927b040eac66a3795ab9e9bc020cbcdc73e87718d7 WatchSource:0}: Error finding container ab345ad15e651ff3a4a74c927b040eac66a3795ab9e9bc020cbcdc73e87718d7: Status 404 returned error can't find the container with id ab345ad15e651ff3a4a74c927b040eac66a3795ab9e9bc020cbcdc73e87718d7 Nov 01 16:18:07.436635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:07.436555 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" event={"ID":"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a","Type":"ContainerStarted","Data":"53df8e7c9ef6a3a84f7093c19069098f7362dd9b1478c53f669fe9e284d97287"} Nov 01 16:18:07.436635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:07.436608 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" event={"ID":"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a","Type":"ContainerStarted","Data":"ab345ad15e651ff3a4a74c927b040eac66a3795ab9e9bc020cbcdc73e87718d7"} Nov 01 16:18:07.453300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:07.453254 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" podStartSLOduration=1.45324105 podStartE2EDuration="1.45324105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:18:07.451556858 +0000 UTC m=+1825.154722798" watchObservedRunningTime="2026-11-01 16:18:07.45324105 +0000 UTC m=+1825.156406986" Nov 01 16:18:07.561887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:07.561850 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:08.417964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:08.417925 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:08.446947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:08.446915 4573 generic.go:356] "Generic (PLEG): container finished" podID="69c5e879-c69f-4c29-b91e-aa7d7cbbb06a" containerID="53df8e7c9ef6a3a84f7093c19069098f7362dd9b1478c53f669fe9e284d97287" exitCode=0 Nov 01 16:18:08.447093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:08.446984 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" event={"ID":"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a","Type":"ContainerDied","Data":"53df8e7c9ef6a3a84f7093c19069098f7362dd9b1478c53f669fe9e284d97287"} Nov 01 16:18:09.153548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.153508 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:09.641094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.641067 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:09.677723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.677692 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-xks-ca\") pod \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " Nov 01 16:18:09.678028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.678006 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-kube-api-access-gdhxx\" (UniqueName: \"kubernetes.io/projected/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-kube-api-access-gdhxx\") pod \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " Nov 01 16:18:09.678164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.678144 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-openshift-ca\") pod \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\" (UID: \"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a\") " Nov 01 16:18:09.678543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.678518 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-openshift-ca" pod "69c5e879-c69f-4c29-b91e-aa7d7cbbb06a" (UID: "69c5e879-c69f-4c29-b91e-aa7d7cbbb06a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:09.678919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.678893 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:09.680028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.680001 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-xks-ca" pod "69c5e879-c69f-4c29-b91e-aa7d7cbbb06a" (UID: "69c5e879-c69f-4c29-b91e-aa7d7cbbb06a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:09.680794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.680731 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-kube-api-access-gdhxx" pod "69c5e879-c69f-4c29-b91e-aa7d7cbbb06a" (UID: "69c5e879-c69f-4c29-b91e-aa7d7cbbb06a"). InnerVolumeSpecName "kube-api-access-gdhxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:09.781224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.781146 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdhxx\" (UniqueName: \"kubernetes.io/projected/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-kube-api-access-gdhxx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:09.781224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.781178 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:09.860926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:09.860880 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:10.465116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:10.465084 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" Nov 01 16:18:10.465116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:10.465108 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px" event={"ID":"69c5e879-c69f-4c29-b91e-aa7d7cbbb06a","Type":"ContainerDied","Data":"ab345ad15e651ff3a4a74c927b040eac66a3795ab9e9bc020cbcdc73e87718d7"} Nov 01 16:18:10.465342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:10.465160 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab345ad15e651ff3a4a74c927b040eac66a3795ab9e9bc020cbcdc73e87718d7" Nov 01 16:18:10.781276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:10.781188 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:11.537552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:11.537516 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:12.289497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:12.289454 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:13.181702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.181671 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:13.829337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.829298 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9"] Nov 01 16:18:13.830301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.830277 4573 cpu_manager.go:414] "containerMap: removing container" podUID="69c5e879-c69f-4c29-b91e-aa7d7cbbb06a" containerName="revoke-keys" Nov 01 16:18:13.830301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.830300 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69c5e879-c69f-4c29-b91e-aa7d7cbbb06a" containerName="revoke-keys" Nov 01 16:18:13.830579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.830563 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="69c5e879-c69f-4c29-b91e-aa7d7cbbb06a" containerName="revoke-keys" Nov 01 16:18:13.834831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.834808 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:13.839033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.839012 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:18:13.839201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.839041 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:18:13.841734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:13.841709 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9"] Nov 01 16:18:14.006075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.006037 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4x9p\" (UniqueName: \"kubernetes.io/projected/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-kube-api-access-k4x9p\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.006391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.006365 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.006521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.006505 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.108554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.108523 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.108713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.108627 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.108779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.108742 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4x9p\" (UniqueName: \"kubernetes.io/projected/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-kube-api-access-k4x9p\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.109309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.109283 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.110975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.110954 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.116635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.116611 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4x9p\" (UniqueName: \"kubernetes.io/projected/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-kube-api-access-k4x9p\") pod \"maas-api-revoke-sub-5143c0db-wpjs9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.148048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.148021 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:14.159196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.159173 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:14.545809 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:18:14.545744 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a48f6f_a1de_4d89_b3b6_4a0fa00631a9.slice/crio-ee765e67b911195e3117cc1a3a3d019e32aa90c59b1815547730ea36d249a93b WatchSource:0}: Error finding container ee765e67b911195e3117cc1a3a3d019e32aa90c59b1815547730ea36d249a93b: Status 404 returned error can't find the container with id ee765e67b911195e3117cc1a3a3d019e32aa90c59b1815547730ea36d249a93b Nov 01 16:18:14.548389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.548361 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9"] Nov 01 16:18:14.792797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:14.792673 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:15.472135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:15.472092 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m"] Nov 01 16:18:15.472563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:15.472294 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" containerID="cri-o://9a7153d1128a488a008af9e546ac49cdfeaf0592f96d0b61f701629e07c4784e" gracePeriod=30 Nov 01 16:18:15.514586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:15.514483 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" event={"ID":"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9","Type":"ContainerStarted","Data":"9035c112329e8abd0f67ca5849bd46c90d6c13516e035ba91aabc733dd755328"} Nov 01 16:18:15.514586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:15.514547 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" event={"ID":"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9","Type":"ContainerStarted","Data":"ee765e67b911195e3117cc1a3a3d019e32aa90c59b1815547730ea36d249a93b"} Nov 01 16:18:15.535970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:15.535901 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" podStartSLOduration=2.535883154 podStartE2EDuration="2.535883154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:18:15.530488062 +0000 UTC m=+1833.233653998" watchObservedRunningTime="2026-11-01 16:18:15.535883154 +0000 UTC m=+1833.239049092" Nov 01 16:18:15.615890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:15.615859 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:15.660554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:15.660525 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq"] Nov 01 16:18:15.660745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:15.660720 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" containerID="cri-o://f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5" gracePeriod=30 Nov 01 16:18:16.418971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:16.418938 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:16.562772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:16.558778 4573 generic.go:356] "Generic (PLEG): container finished" podID="18a48f6f-a1de-4d89-b3b6-4a0fa00631a9" containerID="9035c112329e8abd0f67ca5849bd46c90d6c13516e035ba91aabc733dd755328" exitCode=0 Nov 01 16:18:16.562772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:16.559092 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" event={"ID":"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9","Type":"ContainerDied","Data":"9035c112329e8abd0f67ca5849bd46c90d6c13516e035ba91aabc733dd755328"} Nov 01 16:18:16.685091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:16.685008 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:18:16.685091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:16.685071 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:18:17.167327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.167281 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:17.770020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.769992 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:17.774275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.774249 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-kube-api-access-k4x9p\" (UniqueName: \"kubernetes.io/projected/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-kube-api-access-k4x9p\") pod \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " Nov 01 16:18:17.774481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.774464 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-xks-ca\") pod \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " Nov 01 16:18:17.774568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.774545 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-openshift-ca\") pod \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\" (UID: \"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9\") " Nov 01 16:18:17.774960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.774935 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-openshift-ca" pod "18a48f6f-a1de-4d89-b3b6-4a0fa00631a9" (UID: "18a48f6f-a1de-4d89-b3b6-4a0fa00631a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:17.775346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.775326 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:17.776819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.776796 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-xks-ca" pod "18a48f6f-a1de-4d89-b3b6-4a0fa00631a9" (UID: "18a48f6f-a1de-4d89-b3b6-4a0fa00631a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:17.776914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.776858 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-kube-api-access-k4x9p" pod "18a48f6f-a1de-4d89-b3b6-4a0fa00631a9" (UID: "18a48f6f-a1de-4d89-b3b6-4a0fa00631a9"). InnerVolumeSpecName "kube-api-access-k4x9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:17.876948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.876916 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:17.876948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.876947 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4x9p\" (UniqueName: \"kubernetes.io/projected/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9-kube-api-access-k4x9p\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:17.925641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:17.925605 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:18.585362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.585332 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" Nov 01 16:18:18.585362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.585347 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9" event={"ID":"18a48f6f-a1de-4d89-b3b6-4a0fa00631a9","Type":"ContainerDied","Data":"ee765e67b911195e3117cc1a3a3d019e32aa90c59b1815547730ea36d249a93b"} Nov 01 16:18:18.585593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.585397 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee765e67b911195e3117cc1a3a3d019e32aa90c59b1815547730ea36d249a93b" Nov 01 16:18:18.722105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.722017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:18.844361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.830164 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:18.891775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.887932 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9"] Nov 01 16:18:18.904772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.902326 4573 cpu_manager.go:414] "containerMap: removing container" podUID="18a48f6f-a1de-4d89-b3b6-4a0fa00631a9" containerName="revoke-keys" Nov 01 16:18:18.904772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.902357 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a48f6f-a1de-4d89-b3b6-4a0fa00631a9" containerName="revoke-keys" Nov 01 16:18:18.904772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.902575 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a48f6f-a1de-4d89-b3b6-4a0fa00631a9" containerName="revoke-keys" Nov 01 16:18:18.960786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.960046 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9"] Nov 01 16:18:18.960786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.960215 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:18.967688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.967661 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:18:19.098289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:18.968444 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:18:19.098289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.026073 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj8rw\" (UniqueName: \"kubernetes.io/projected/366a6a5b-0129-4384-b619-5a04f775ee00-kube-api-access-qj8rw\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.098542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.098353 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.098727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.098706 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.199961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.199928 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.200167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.200060 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj8rw\" (UniqueName: \"kubernetes.io/projected/366a6a5b-0129-4384-b619-5a04f775ee00-kube-api-access-qj8rw\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.200167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.200081 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.200719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.200662 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.202658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.202637 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.208034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.208012 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj8rw\" (UniqueName: \"kubernetes.io/projected/366a6a5b-0129-4384-b619-5a04f775ee00-kube-api-access-qj8rw\") pod \"maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.289264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.289171 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:19.575765 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:18:19.575724 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366a6a5b_0129_4384_b619_5a04f775ee00.slice/crio-ed51f45d810e73e773487f54a85c9471ba9e0a7842e4d9a9a38b9ef4a137d0fb WatchSource:0}: Error finding container ed51f45d810e73e773487f54a85c9471ba9e0a7842e4d9a9a38b9ef4a137d0fb: Status 404 returned error can't find the container with id ed51f45d810e73e773487f54a85c9471ba9e0a7842e4d9a9a38b9ef4a137d0fb Nov 01 16:18:19.576036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.576013 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9"] Nov 01 16:18:19.594736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.594706 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" event={"ID":"366a6a5b-0129-4384-b619-5a04f775ee00","Type":"ContainerStarted","Data":"ed51f45d810e73e773487f54a85c9471ba9e0a7842e4d9a9a38b9ef4a137d0fb"} Nov 01 16:18:19.869502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:19.869469 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:20.599600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.599555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:20.606190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.606159 4573 generic.go:356] "Generic (PLEG): container finished" podID="366a6a5b-0129-4384-b619-5a04f775ee00" containerID="691093e8b1615079c77ecbacbe72a3ba7cfd235027666cee974c386ab1fb31d2" exitCode=0 Nov 01 16:18:20.606454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.606256 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" event={"ID":"366a6a5b-0129-4384-b619-5a04f775ee00","Type":"ContainerDied","Data":"691093e8b1615079c77ecbacbe72a3ba7cfd235027666cee974c386ab1fb31d2"} Nov 01 16:18:20.608395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.608372 4573 generic.go:356] "Generic (PLEG): container finished" podID="9e849359-8086-44e6-8a61-b879af942f35" containerID="9a7153d1128a488a008af9e546ac49cdfeaf0592f96d0b61f701629e07c4784e" exitCode=0 Nov 01 16:18:20.608498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.608448 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" event={"ID":"9e849359-8086-44e6-8a61-b879af942f35","Type":"ContainerDied","Data":"9a7153d1128a488a008af9e546ac49cdfeaf0592f96d0b61f701629e07c4784e"} Nov 01 16:18:20.874276 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:18:20.874232 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366a6a5b_0129_4384_b619_5a04f775ee00.slice/crio-conmon-691093e8b1615079c77ecbacbe72a3ba7cfd235027666cee974c386ab1fb31d2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c8337d_1c52_456e_b8e7_8afcd6f30eec.slice/crio-f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:18:20.886413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.886387 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:18:20.935259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935218 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-certs\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.935426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935304 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-socket\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.935426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935405 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-credential-socket\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.935545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935448 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-data\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.935851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935604 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-envoy\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.935851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935651 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-kube-api-access-rd2gr\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-kube-api-access-rd2gr\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.935851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935717 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e849359-8086-44e6-8a61-b879af942f35-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e849359-8086-44e6-8a61-b879af942f35-istio-podinfo\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.936054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935892 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e849359-8086-44e6-8a61-b879af942f35-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e849359-8086-44e6-8a61-b879af942f35-istiod-ca-cert\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.936054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.935995 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-istio-token\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-istio-token\") pod \"9e849359-8086-44e6-8a61-b879af942f35\" (UID: \"9e849359-8086-44e6-8a61-b879af942f35\") " Nov 01 16:18:20.936385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.936209 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-credential-socket" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:20.936697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.936510 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-socket" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:20.936843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.936825 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-data" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:20.936997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.936971 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-certs" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:20.937402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.937379 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:20.937513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.937501 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:20.937603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.937592 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:20.937684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.937673 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:20.945852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.945810 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e849359-8086-44e6-8a61-b879af942f35-istiod-ca-cert" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:20.949202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.949171 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-envoy" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:20.953415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.952059 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-kube-api-access-rd2gr" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "kube-api-access-rd2gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:20.953415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.952100 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e849359-8086-44e6-8a61-b879af942f35-istio-podinfo" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:20.955256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:20.955234 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-istio-token" pod "9e849359-8086-44e6-8a61-b879af942f35" (UID: "9e849359-8086-44e6-8a61-b879af942f35"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.039138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.039103 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e849359-8086-44e6-8a61-b879af942f35-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.039318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.039160 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2gr\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-kube-api-access-rd2gr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.039318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.039173 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e849359-8086-44e6-8a61-b879af942f35-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.039318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.039183 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e849359-8086-44e6-8a61-b879af942f35-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.039318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.039196 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e849359-8086-44e6-8a61-b879af942f35-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.040886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.040856 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:18:21.139833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.139792 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-data\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.140049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.139862 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-kube-api-access-zxwj4\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-kube-api-access-zxwj4\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.140049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.139953 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-certs\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.140170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140092 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-socket\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.140170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140127 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-data" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.140286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140220 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-credential-socket\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.140341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140297 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-envoy\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.140341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140293 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-certs" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.140450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140376 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-socket" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.140450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140405 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-podinfo\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.140568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140458 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-token\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-token\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.140568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140481 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-credential-socket" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.140568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140508 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istiod-ca-cert\") pod \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\" (UID: \"63c8337d-1c52-456e-b8e7-8afcd6f30eec\") " Nov 01 16:18:21.141000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.140974 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istiod-ca-cert" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:21.141303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.141276 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.141411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.141308 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.141411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.141323 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.141411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.141338 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.141411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.141352 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.143140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.143108 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-token" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.143316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.143169 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-envoy" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:21.143316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.143227 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-kube-api-access-zxwj4" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "kube-api-access-zxwj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:21.143668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.143644 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-podinfo" pod "63c8337d-1c52-456e-b8e7-8afcd6f30eec" (UID: "63c8337d-1c52-456e-b8e7-8afcd6f30eec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:21.242533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.242501 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.242533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.242529 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.242828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.242552 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.242828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.242562 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwj4\" (UniqueName: \"kubernetes.io/projected/63c8337d-1c52-456e-b8e7-8afcd6f30eec-kube-api-access-zxwj4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.595053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.595014 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:21.622261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.622224 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" event={"ID":"9e849359-8086-44e6-8a61-b879af942f35","Type":"ContainerDied","Data":"9aac7740a4257614fd4e135967c41d920dfd4825b06273903fe33b4e4e5f161f"} Nov 01 16:18:21.622401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.622301 4573 scope.go:122] "RemoveContainer" containerID="9a7153d1128a488a008af9e546ac49cdfeaf0592f96d0b61f701629e07c4784e" Nov 01 16:18:21.622502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.622485 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m" Nov 01 16:18:21.624614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.624591 4573 generic.go:356] "Generic (PLEG): container finished" podID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerID="f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5" exitCode=0 Nov 01 16:18:21.624921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.624903 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" Nov 01 16:18:21.625366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.625306 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" event={"ID":"63c8337d-1c52-456e-b8e7-8afcd6f30eec","Type":"ContainerDied","Data":"f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5"} Nov 01 16:18:21.625474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.625375 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq" event={"ID":"63c8337d-1c52-456e-b8e7-8afcd6f30eec","Type":"ContainerDied","Data":"cdec607e63f53adf366adabc709d777518a57cce1b9aadd39a2ffbe3880423a8"} Nov 01 16:18:21.637525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.637473 4573 scope.go:122] "RemoveContainer" containerID="f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5" Nov 01 16:18:21.649195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.649169 4573 scope.go:122] "RemoveContainer" containerID="f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5" Nov 01 16:18:21.649647 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:18:21.649620 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5\": container with ID starting with f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5 not found: ID does not exist" containerID="f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5" Nov 01 16:18:21.649795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.649658 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5"} err="failed to get container status \"f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5\": rpc error: code = NotFound desc = could not find container \"f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5\": container with ID starting with f5a3198cca6d9a44cb380bdd6700c19fe1452d265516759fb1ce94f6e2f929c5 not found: ID does not exist" Nov 01 16:18:21.654726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.654653 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq"] Nov 01 16:18:21.660671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.660648 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-76lsq"] Nov 01 16:18:21.678572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.678540 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m"] Nov 01 16:18:21.683008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.682988 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-g8x6m"] Nov 01 16:18:21.896915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.896889 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:21.995673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.995644 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-openshift-ca\") pod \"366a6a5b-0129-4384-b619-5a04f775ee00\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " Nov 01 16:18:21.995899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.995775 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-xks-ca\") pod \"366a6a5b-0129-4384-b619-5a04f775ee00\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " Nov 01 16:18:21.995999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.995941 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/366a6a5b-0129-4384-b619-5a04f775ee00-kube-api-access-qj8rw\" (UniqueName: \"kubernetes.io/projected/366a6a5b-0129-4384-b619-5a04f775ee00-kube-api-access-qj8rw\") pod \"366a6a5b-0129-4384-b619-5a04f775ee00\" (UID: \"366a6a5b-0129-4384-b619-5a04f775ee00\") " Nov 01 16:18:21.996065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.996040 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-openshift-ca" pod "366a6a5b-0129-4384-b619-5a04f775ee00" (UID: "366a6a5b-0129-4384-b619-5a04f775ee00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:21.996450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.996430 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:21.997996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.997974 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-xks-ca" pod "366a6a5b-0129-4384-b619-5a04f775ee00" (UID: "366a6a5b-0129-4384-b619-5a04f775ee00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:21.998094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:21.998068 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366a6a5b-0129-4384-b619-5a04f775ee00-kube-api-access-qj8rw" pod "366a6a5b-0129-4384-b619-5a04f775ee00" (UID: "366a6a5b-0129-4384-b619-5a04f775ee00"). InnerVolumeSpecName "kube-api-access-qj8rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:22.097785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:22.097740 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj8rw\" (UniqueName: \"kubernetes.io/projected/366a6a5b-0129-4384-b619-5a04f775ee00-kube-api-access-qj8rw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:22.097785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:22.097782 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/366a6a5b-0129-4384-b619-5a04f775ee00-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:22.380939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:22.380900 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:22.536523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:22.536489 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" path="/var/lib/kubelet/pods/63c8337d-1c52-456e-b8e7-8afcd6f30eec/volumes" Nov 01 16:18:22.537317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:22.537291 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e849359-8086-44e6-8a61-b879af942f35" path="/var/lib/kubelet/pods/9e849359-8086-44e6-8a61-b879af942f35/volumes" Nov 01 16:18:22.636454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:22.636373 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" event={"ID":"366a6a5b-0129-4384-b619-5a04f775ee00","Type":"ContainerDied","Data":"ed51f45d810e73e773487f54a85c9471ba9e0a7842e4d9a9a38b9ef4a137d0fb"} Nov 01 16:18:22.636454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:22.636436 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed51f45d810e73e773487f54a85c9471ba9e0a7842e4d9a9a38b9ef4a137d0fb" Nov 01 16:18:22.636454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:22.636387 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9" Nov 01 16:18:23.135797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:23.135767 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:23.878032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:23.877933 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:24.366033 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:18:24.365997 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:18:24.652149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:24.652054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:25.494092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:25.494056 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:26.386715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:26.386665 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:27.419730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:27.419689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:28.121367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:28.121318 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:29.092203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:29.092160 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:29.190577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:29.190548 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:30.104189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:30.104156 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:30.205363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:30.205330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:30.317775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:30.309593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:30.923503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:30.923476 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:31.586561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:31.586530 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:31.696777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:31.693271 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:32.210886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.210852 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:32.420318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.420278 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:32.590212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.590131 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:32.725729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.725698 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7"] Nov 01 16:18:32.727234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727207 4573 cpu_manager.go:414] "containerMap: removing container" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" Nov 01 16:18:32.727234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727234 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" Nov 01 16:18:32.727477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727266 4573 cpu_manager.go:414] "containerMap: removing container" podUID="366a6a5b-0129-4384-b619-5a04f775ee00" containerName="revoke-keys" Nov 01 16:18:32.727477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727275 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="366a6a5b-0129-4384-b619-5a04f775ee00" containerName="revoke-keys" Nov 01 16:18:32.727477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727336 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" Nov 01 16:18:32.727477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727344 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" Nov 01 16:18:32.727684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727573 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e849359-8086-44e6-8a61-b879af942f35" containerName="istio-proxy" Nov 01 16:18:32.727684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727599 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c8337d-1c52-456e-b8e7-8afcd6f30eec" containerName="istio-proxy" Nov 01 16:18:32.727684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.727610 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="366a6a5b-0129-4384-b619-5a04f775ee00" containerName="revoke-keys" Nov 01 16:18:32.737541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.737507 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:32.737924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.737893 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.751420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.737725 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7"] Nov 01 16:18:32.751420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.744267 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:18:32.751420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.744401 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:18:32.870198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.870165 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnz9k\" (UniqueName: \"kubernetes.io/projected/8dea3b94-4699-48b5-93ab-ad2c031557d0-kube-api-access-rnz9k\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.870482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.870457 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.870546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.870518 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.972533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.972500 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnz9k\" (UniqueName: \"kubernetes.io/projected/8dea3b94-4699-48b5-93ab-ad2c031557d0-kube-api-access-rnz9k\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.972737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.972718 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.972844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.972821 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.973441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.973417 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.975037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.975019 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:32.983867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:32.983841 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnz9k\" (UniqueName: \"kubernetes.io/projected/8dea3b94-4699-48b5-93ab-ad2c031557d0-kube-api-access-rnz9k\") pod \"maas-api-revoke-sub-a6213a38-n7zv7\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:33.078912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:33.078884 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:33.383902 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:18:33.383874 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dea3b94_4699_48b5_93ab_ad2c031557d0.slice/crio-d8ed17ef45f15fea8053c50f4da68592d9e043bec721bb965883d8c9409884f8 WatchSource:0}: Error finding container d8ed17ef45f15fea8053c50f4da68592d9e043bec721bb965883d8c9409884f8: Status 404 returned error can't find the container with id d8ed17ef45f15fea8053c50f4da68592d9e043bec721bb965883d8c9409884f8 Nov 01 16:18:33.386559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:33.386446 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7"] Nov 01 16:18:33.676829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:33.676626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:33.772404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:33.772368 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" event={"ID":"8dea3b94-4699-48b5-93ab-ad2c031557d0","Type":"ContainerStarted","Data":"d8ed17ef45f15fea8053c50f4da68592d9e043bec721bb965883d8c9409884f8"} Nov 01 16:18:34.428220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:34.428168 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:34.782423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:34.782385 4573 generic.go:356] "Generic (PLEG): container finished" podID="8dea3b94-4699-48b5-93ab-ad2c031557d0" containerID="0370bb85c298779f85078735d8bd8dd6cdf970d0f6f36b1632a527daefca4ef2" exitCode=0 Nov 01 16:18:34.782999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:34.782472 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" event={"ID":"8dea3b94-4699-48b5-93ab-ad2c031557d0","Type":"ContainerDied","Data":"0370bb85c298779f85078735d8bd8dd6cdf970d0f6f36b1632a527daefca4ef2"} Nov 01 16:18:35.434640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:35.434607 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:35.984844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:35.984817 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:36.036315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.036286 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-xks-ca\") pod \"8dea3b94-4699-48b5-93ab-ad2c031557d0\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " Nov 01 16:18:36.036533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.036518 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dea3b94-4699-48b5-93ab-ad2c031557d0-kube-api-access-rnz9k\" (UniqueName: \"kubernetes.io/projected/8dea3b94-4699-48b5-93ab-ad2c031557d0-kube-api-access-rnz9k\") pod \"8dea3b94-4699-48b5-93ab-ad2c031557d0\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " Nov 01 16:18:36.036602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.036584 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-openshift-ca\") pod \"8dea3b94-4699-48b5-93ab-ad2c031557d0\" (UID: \"8dea3b94-4699-48b5-93ab-ad2c031557d0\") " Nov 01 16:18:36.037020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.036981 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-openshift-ca" pod "8dea3b94-4699-48b5-93ab-ad2c031557d0" (UID: "8dea3b94-4699-48b5-93ab-ad2c031557d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:36.037498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.037475 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:36.038495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.038471 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-xks-ca" pod "8dea3b94-4699-48b5-93ab-ad2c031557d0" (UID: "8dea3b94-4699-48b5-93ab-ad2c031557d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:36.039101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.039080 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dea3b94-4699-48b5-93ab-ad2c031557d0-kube-api-access-rnz9k" pod "8dea3b94-4699-48b5-93ab-ad2c031557d0" (UID: "8dea3b94-4699-48b5-93ab-ad2c031557d0"). InnerVolumeSpecName "kube-api-access-rnz9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:36.138907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.138880 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dea3b94-4699-48b5-93ab-ad2c031557d0-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:36.138907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.138906 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnz9k\" (UniqueName: \"kubernetes.io/projected/8dea3b94-4699-48b5-93ab-ad2c031557d0-kube-api-access-rnz9k\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:36.262884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.262853 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:36.801311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.801235 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" Nov 01 16:18:36.801311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.801256 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7" event={"ID":"8dea3b94-4699-48b5-93ab-ad2c031557d0","Type":"ContainerDied","Data":"d8ed17ef45f15fea8053c50f4da68592d9e043bec721bb965883d8c9409884f8"} Nov 01 16:18:36.801311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.801302 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8ed17ef45f15fea8053c50f4da68592d9e043bec721bb965883d8c9409884f8" Nov 01 16:18:36.958283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:36.958243 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:37.772299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:37.772242 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:38.388914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:38.388875 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:38.533028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:38.532991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:38.646823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:38.646119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:38.966196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:38.965876 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9"] Nov 01 16:18:38.970949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:38.970920 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f5dd93-h9jz9"] Nov 01 16:18:39.212220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:39.212188 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:39.325825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:39.325648 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:40.055167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:40.055124 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:40.461219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:40.461182 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="366a6a5b-0129-4384-b619-5a04f775ee00" path="/var/lib/kubelet/pods/366a6a5b-0129-4384-b619-5a04f775ee00/volumes" Nov 01 16:18:40.775433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:40.775341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:40.910970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:40.910936 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:41.013399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:41.013366 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:41.463803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:41.463734 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:42.222943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:42.222909 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:43.002058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:43.002016 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:43.666860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:43.666826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:44.426722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:44.426687 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:45.010565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.010530 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:45.393988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.393952 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8"] Nov 01 16:18:45.394822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.394801 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8dea3b94-4699-48b5-93ab-ad2c031557d0" containerName="revoke-keys" Nov 01 16:18:45.394822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.394819 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dea3b94-4699-48b5-93ab-ad2c031557d0" containerName="revoke-keys" Nov 01 16:18:45.395016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.395003 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dea3b94-4699-48b5-93ab-ad2c031557d0" containerName="revoke-keys" Nov 01 16:18:45.400167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.400143 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.407145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.403451 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:18:45.407145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.404150 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:18:45.407368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.407343 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8"] Nov 01 16:18:45.429535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.429496 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.430034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.429641 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.430034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.429765 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stmdr\" (UniqueName: \"kubernetes.io/projected/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-kube-api-access-stmdr\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.531349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.531303 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.531537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.531417 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.531537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.531507 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stmdr\" (UniqueName: \"kubernetes.io/projected/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-kube-api-access-stmdr\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.532225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.532196 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.534387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.534362 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.539471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.539440 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stmdr\" (UniqueName: \"kubernetes.io/projected/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-kube-api-access-stmdr\") pod \"maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:45.653336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.653255 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:45.723782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:45.722086 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:46.060489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:46.060159 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8"] Nov 01 16:18:46.508709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:46.508668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:46.608156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:46.608128 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:46.791240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:46.791148 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s"] Nov 01 16:18:46.791389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:46.791363 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" containerID="cri-o://815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d" gracePeriod=30 Nov 01 16:18:46.808542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:46.808513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:47.115317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.113370 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" event={"ID":"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc","Type":"ContainerStarted","Data":"8751bba9f070f31f958a9da22036142d02307a22cfaec9abcbadf213807bd830"} Nov 01 16:18:47.115317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.113437 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" event={"ID":"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc","Type":"ContainerStarted","Data":"af8b3c312a97159808e568cf30a3e86cddf2296526cc41d86b65157c563f39c1"} Nov 01 16:18:47.394113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.394004 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" podStartSLOduration=2.3939893039999998 podStartE2EDuration="2.393989304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:18:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:18:47.158193465 +0000 UTC m=+1864.861359442" watchObservedRunningTime="2026-11-01 16:18:47.393989304 +0000 UTC m=+1865.097155240" Nov 01 16:18:47.398533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.398504 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf"] Nov 01 16:18:47.404801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.404780 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.408328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.408285 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmhk9\" (UniqueName: \"kubernetes.io/projected/cd851cef-caac-4d44-bba5-956cd06852de-kube-api-access-qmhk9\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.408702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.408494 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.408702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.408654 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.412319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.412294 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf"] Nov 01 16:18:47.457382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.457353 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:47.515843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.515800 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.548404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.548370 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.549351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.549321 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmhk9\" (UniqueName: \"kubernetes.io/projected/cd851cef-caac-4d44-bba5-956cd06852de-kube-api-access-qmhk9\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.561795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.523414 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.573539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.573479 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmhk9\" (UniqueName: \"kubernetes.io/projected/cd851cef-caac-4d44-bba5-956cd06852de-kube-api-access-qmhk9\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.588267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.588234 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-375350-lncvf\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.724891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.724799 4573 scope.go:122] "RemoveContainer" containerID="31ad18d61240b3be32349091bde6253b66b8d50cfa8c4e3d04ba38cf4efba3d1" Nov 01 16:18:47.731223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.731200 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:47.737799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:47.737738 4573 scope.go:122] "RemoveContainer" containerID="fba99e310295e36aa3b3647166538e5953dde89f8387788db1141b1b8d6620e2" Nov 01 16:18:48.035710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:48.035663 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf"] Nov 01 16:18:48.036996 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:18:48.036941 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd851cef_caac_4d44_bba5_956cd06852de.slice/crio-97a067c6216291fe6f155aae4982c16aed6704b8fe3086e9ec3b15e9a59e6703 WatchSource:0}: Error finding container 97a067c6216291fe6f155aae4982c16aed6704b8fe3086e9ec3b15e9a59e6703: Status 404 returned error can't find the container with id 97a067c6216291fe6f155aae4982c16aed6704b8fe3086e9ec3b15e9a59e6703 Nov 01 16:18:48.122427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:48.122393 4573 generic.go:356] "Generic (PLEG): container finished" podID="7eb9defd-7a5a-442c-a649-e3cc8c4c06bc" containerID="8751bba9f070f31f958a9da22036142d02307a22cfaec9abcbadf213807bd830" exitCode=0 Nov 01 16:18:48.122603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:48.122472 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" event={"ID":"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc","Type":"ContainerDied","Data":"8751bba9f070f31f958a9da22036142d02307a22cfaec9abcbadf213807bd830"} Nov 01 16:18:48.123885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:48.123863 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" event={"ID":"cd851cef-caac-4d44-bba5-956cd06852de","Type":"ContainerStarted","Data":"97a067c6216291fe6f155aae4982c16aed6704b8fe3086e9ec3b15e9a59e6703"} Nov 01 16:18:48.355729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:48.355647 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:49.087275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.087188 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:49.133705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.133661 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" event={"ID":"cd851cef-caac-4d44-bba5-956cd06852de","Type":"ContainerStarted","Data":"beb2cf313cc193cd6de95313650df2dbe88e157e699052bf931cc1c6a84ed4df"} Nov 01 16:18:49.207840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.201154 4573 patch_prober.go:29] interesting pod/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:18:49.207840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.201233 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:18:49.219426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.218493 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" podStartSLOduration=2.218473217 podStartE2EDuration="2.218473217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:18:49.177197764 +0000 UTC m=+1866.880363700" watchObservedRunningTime="2026-11-01 16:18:49.218473217 +0000 UTC m=+1866.921639154" Nov 01 16:18:49.400996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.400843 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:49.506609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.506575 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-kube-api-access-stmdr\" (UniqueName: \"kubernetes.io/projected/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-kube-api-access-stmdr\") pod \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " Nov 01 16:18:49.506819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.506669 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-openshift-ca\") pod \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " Nov 01 16:18:49.506819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.506735 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-xks-ca\") pod \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\" (UID: \"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc\") " Nov 01 16:18:49.507104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.507070 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-openshift-ca" pod "7eb9defd-7a5a-442c-a649-e3cc8c4c06bc" (UID: "7eb9defd-7a5a-442c-a649-e3cc8c4c06bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:49.507512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.507492 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:49.508908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.508890 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-xks-ca" pod "7eb9defd-7a5a-442c-a649-e3cc8c4c06bc" (UID: "7eb9defd-7a5a-442c-a649-e3cc8c4c06bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:49.509093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.509074 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-kube-api-access-stmdr" pod "7eb9defd-7a5a-442c-a649-e3cc8c4c06bc" (UID: "7eb9defd-7a5a-442c-a649-e3cc8c4c06bc"). InnerVolumeSpecName "kube-api-access-stmdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:49.608731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.608695 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stmdr\" (UniqueName: \"kubernetes.io/projected/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-kube-api-access-stmdr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:49.608731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.608728 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:49.861907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:49.861850 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:50.144208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:50.144119 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" Nov 01 16:18:50.144208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:50.144150 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8" event={"ID":"7eb9defd-7a5a-442c-a649-e3cc8c4c06bc","Type":"ContainerDied","Data":"af8b3c312a97159808e568cf30a3e86cddf2296526cc41d86b65157c563f39c1"} Nov 01 16:18:50.144208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:50.144199 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af8b3c312a97159808e568cf30a3e86cddf2296526cc41d86b65157c563f39c1" Nov 01 16:18:50.146096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:50.146065 4573 generic.go:356] "Generic (PLEG): container finished" podID="cd851cef-caac-4d44-bba5-956cd06852de" containerID="beb2cf313cc193cd6de95313650df2dbe88e157e699052bf931cc1c6a84ed4df" exitCode=0 Nov 01 16:18:50.146243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:50.146145 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" event={"ID":"cd851cef-caac-4d44-bba5-956cd06852de","Type":"ContainerDied","Data":"beb2cf313cc193cd6de95313650df2dbe88e157e699052bf931cc1c6a84ed4df"} Nov 01 16:18:50.641061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:50.641024 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:51.242570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.242467 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88"] Nov 01 16:18:51.243861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.243838 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7eb9defd-7a5a-442c-a649-e3cc8c4c06bc" containerName="revoke-keys" Nov 01 16:18:51.243861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.243861 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eb9defd-7a5a-442c-a649-e3cc8c4c06bc" containerName="revoke-keys" Nov 01 16:18:51.244111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.244093 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eb9defd-7a5a-442c-a649-e3cc8c4c06bc" containerName="revoke-keys" Nov 01 16:18:51.248829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.248805 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.253341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.253317 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88"] Nov 01 16:18:51.343547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.343519 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:51.352961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.352935 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:51.361485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.360863 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx8nq\" (UniqueName: \"kubernetes.io/projected/f72f3e0c-6521-441d-8c54-850fb767d5d8-kube-api-access-bx8nq\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.361485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.360916 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.361485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.361280 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.480853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.480114 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-openshift-ca\") pod \"cd851cef-caac-4d44-bba5-956cd06852de\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " Nov 01 16:18:51.482175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.481803 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-openshift-ca" pod "cd851cef-caac-4d44-bba5-956cd06852de" (UID: "cd851cef-caac-4d44-bba5-956cd06852de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:51.482175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.481741 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-xks-ca\") pod \"cd851cef-caac-4d44-bba5-956cd06852de\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " Nov 01 16:18:51.482432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.482410 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd851cef-caac-4d44-bba5-956cd06852de-kube-api-access-qmhk9\" (UniqueName: \"kubernetes.io/projected/cd851cef-caac-4d44-bba5-956cd06852de-kube-api-access-qmhk9\") pod \"cd851cef-caac-4d44-bba5-956cd06852de\" (UID: \"cd851cef-caac-4d44-bba5-956cd06852de\") " Nov 01 16:18:51.488682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.488652 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.488954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.488928 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx8nq\" (UniqueName: \"kubernetes.io/projected/f72f3e0c-6521-441d-8c54-850fb767d5d8-kube-api-access-bx8nq\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.489034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.488981 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.489511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.489491 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:51.491082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.491047 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.493123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.493106 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.494793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.494765 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd851cef-caac-4d44-bba5-956cd06852de-kube-api-access-qmhk9" pod "cd851cef-caac-4d44-bba5-956cd06852de" (UID: "cd851cef-caac-4d44-bba5-956cd06852de"). InnerVolumeSpecName "kube-api-access-qmhk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:51.495125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.495105 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-xks-ca" pod "cd851cef-caac-4d44-bba5-956cd06852de" (UID: "cd851cef-caac-4d44-bba5-956cd06852de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:51.498419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.498394 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx8nq\" (UniqueName: \"kubernetes.io/projected/f72f3e0c-6521-441d-8c54-850fb767d5d8-kube-api-access-bx8nq\") pod \"maas-api-revoke-sub-09a26172-f7k88\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.569470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.569438 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:51.591429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.591400 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd851cef-caac-4d44-bba5-956cd06852de-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:51.591542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.591433 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmhk9\" (UniqueName: \"kubernetes.io/projected/cd851cef-caac-4d44-bba5-956cd06852de-kube-api-access-qmhk9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:51.856774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:51.856729 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88"] Nov 01 16:18:51.864689 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:18:51.864664 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72f3e0c_6521_441d_8c54_850fb767d5d8.slice/crio-8e3bb3cb0eefa347cb8698d2be0093a66cb7de7b22caa92dba6cf66754f0ffbb WatchSource:0}: Error finding container 8e3bb3cb0eefa347cb8698d2be0093a66cb7de7b22caa92dba6cf66754f0ffbb: Status 404 returned error can't find the container with id 8e3bb3cb0eefa347cb8698d2be0093a66cb7de7b22caa92dba6cf66754f0ffbb Nov 01 16:18:52.039520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.039478 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:52.061689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.061628 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:18:52.193612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.193571 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" event={"ID":"cd851cef-caac-4d44-bba5-956cd06852de","Type":"ContainerDied","Data":"97a067c6216291fe6f155aae4982c16aed6704b8fe3086e9ec3b15e9a59e6703"} Nov 01 16:18:52.193879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.193645 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97a067c6216291fe6f155aae4982c16aed6704b8fe3086e9ec3b15e9a59e6703" Nov 01 16:18:52.193879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.193731 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf" Nov 01 16:18:52.195714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.195681 4573 generic.go:356] "Generic (PLEG): container finished" podID="751315de-016b-4107-9b93-c3d4d139b9c0" containerID="815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d" exitCode=0 Nov 01 16:18:52.195933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.195914 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" Nov 01 16:18:52.196031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.195917 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" event={"ID":"751315de-016b-4107-9b93-c3d4d139b9c0","Type":"ContainerDied","Data":"815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d"} Nov 01 16:18:52.196103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.196070 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s" event={"ID":"751315de-016b-4107-9b93-c3d4d139b9c0","Type":"ContainerDied","Data":"475e2e0d4c5b88144e2520fc119bc1804b87c194e7346ba4a936659b959659db"} Nov 01 16:18:52.196164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.196108 4573 scope.go:122] "RemoveContainer" containerID="815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d" Nov 01 16:18:52.198122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.197743 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" event={"ID":"f72f3e0c-6521-441d-8c54-850fb767d5d8","Type":"ContainerStarted","Data":"8e3bb3cb0eefa347cb8698d2be0093a66cb7de7b22caa92dba6cf66754f0ffbb"} Nov 01 16:18:52.247017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.246977 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-credential-socket\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.247445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.247028 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-envoy\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.247445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.247194 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-socket\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.247445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.247251 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-istio-token\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-istio-token\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.247445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.247302 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/751315de-016b-4107-9b93-c3d4d139b9c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751315de-016b-4107-9b93-c3d4d139b9c0-istio-podinfo\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.247445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.247364 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-kube-api-access-27rx7\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-kube-api-access-27rx7\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.247736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.247481 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/751315de-016b-4107-9b93-c3d4d139b9c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751315de-016b-4107-9b93-c3d4d139b9c0-istiod-ca-cert\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.247736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.247556 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-data\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.247736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.247632 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-certs\") pod \"751315de-016b-4107-9b93-c3d4d139b9c0\" (UID: \"751315de-016b-4107-9b93-c3d4d139b9c0\") " Nov 01 16:18:52.249116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.249049 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-certs" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.249232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.249179 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-credential-socket" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.249426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.249404 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-socket" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.249523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.249418 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751315de-016b-4107-9b93-c3d4d139b9c0-istiod-ca-cert" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:52.249642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.249604 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-data" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.265892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.265861 4573 scope.go:122] "RemoveContainer" containerID="815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d" Nov 01 16:18:52.266611 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:18:52.266582 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d\": container with ID starting with 815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d not found: ID does not exist" containerID="815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d" Nov 01 16:18:52.266667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.266630 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d"} err="failed to get container status \"815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d\": rpc error: code = NotFound desc = could not find container \"815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d\": container with ID starting with 815882e62fe01a75379204e57fcd5ee53597df91e021876d83be1ec9a4b3af3d not found: ID does not exist" Nov 01 16:18:52.270557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.270523 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-envoy" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:18:52.272314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.272288 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/751315de-016b-4107-9b93-c3d4d139b9c0-istio-podinfo" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:18:52.272461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.272305 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-istio-token" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:52.274301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.274279 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-kube-api-access-27rx7" pod "751315de-016b-4107-9b93-c3d4d139b9c0" (UID: "751315de-016b-4107-9b93-c3d4d139b9c0"). InnerVolumeSpecName "kube-api-access-27rx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:52.351347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351310 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.351481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351350 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.351481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351366 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.351481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351381 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.351481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351395 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/751315de-016b-4107-9b93-c3d4d139b9c0-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.351481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351410 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27rx7\" (UniqueName: \"kubernetes.io/projected/751315de-016b-4107-9b93-c3d4d139b9c0-kube-api-access-27rx7\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.351481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351425 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/751315de-016b-4107-9b93-c3d4d139b9c0-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.351481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351438 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.351481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.351451 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/751315de-016b-4107-9b93-c3d4d139b9c0-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:52.517266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.517231 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s"] Nov 01 16:18:52.522111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.522086 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f5dd93-openshift-default-86c6c56f5c-6pc6s"] Nov 01 16:18:52.858051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.858017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:52.962999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.957919 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c"] Nov 01 16:18:52.962999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.959390 4573 cpu_manager.go:414] "containerMap: removing container" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" Nov 01 16:18:52.962999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.959415 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" Nov 01 16:18:52.962999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.959460 4573 cpu_manager.go:414] "containerMap: removing container" podUID="cd851cef-caac-4d44-bba5-956cd06852de" containerName="revoke-keys" Nov 01 16:18:52.962999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.959467 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd851cef-caac-4d44-bba5-956cd06852de" containerName="revoke-keys" Nov 01 16:18:52.970398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.969717 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd851cef-caac-4d44-bba5-956cd06852de" containerName="revoke-keys" Nov 01 16:18:52.970398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.969793 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" containerName="istio-proxy" Nov 01 16:18:52.989463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.980594 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:52.989463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:52.986152 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c"] Nov 01 16:18:53.071957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.071916 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50cf165-0412-4978-9947-392132261849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.072146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.071977 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50cf165-0412-4978-9947-392132261849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.072146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.072015 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k89b\" (UniqueName: \"kubernetes.io/projected/b50cf165-0412-4978-9947-392132261849-kube-api-access-7k89b\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.173342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.173263 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50cf165-0412-4978-9947-392132261849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.173342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.173308 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50cf165-0412-4978-9947-392132261849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.173342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.173336 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k89b\" (UniqueName: \"kubernetes.io/projected/b50cf165-0412-4978-9947-392132261849-kube-api-access-7k89b\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.174062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.174023 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50cf165-0412-4978-9947-392132261849-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.175716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.175698 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50cf165-0412-4978-9947-392132261849-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.180643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.180621 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k89b\" (UniqueName: \"kubernetes.io/projected/b50cf165-0412-4978-9947-392132261849-kube-api-access-7k89b\") pod \"maas-api-revoke-sub-18437e32-x5q9c\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.209455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.209425 4573 generic.go:356] "Generic (PLEG): container finished" podID="f72f3e0c-6521-441d-8c54-850fb767d5d8" containerID="c24bbbf925a79477aef1213d7a4c7c70ac464fcdedaf93b53dedd7252cc33e66" exitCode=0 Nov 01 16:18:53.209596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.209505 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" event={"ID":"f72f3e0c-6521-441d-8c54-850fb767d5d8","Type":"ContainerDied","Data":"c24bbbf925a79477aef1213d7a4c7c70ac464fcdedaf93b53dedd7252cc33e66"} Nov 01 16:18:53.336830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.336798 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:53.620899 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:18:53.620865 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50cf165_0412_4978_9947_392132261849.slice/crio-4856d778e1916ebbe8cf1c4d3904be475c43ff674bb4f6e906bfe1ea164a9409 WatchSource:0}: Error finding container 4856d778e1916ebbe8cf1c4d3904be475c43ff674bb4f6e906bfe1ea164a9409: Status 404 returned error can't find the container with id 4856d778e1916ebbe8cf1c4d3904be475c43ff674bb4f6e906bfe1ea164a9409 Nov 01 16:18:53.621422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.621139 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c"] Nov 01 16:18:53.621422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:53.621356 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:54.219452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.219350 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" event={"ID":"b50cf165-0412-4978-9947-392132261849","Type":"ContainerStarted","Data":"b2c8fbc46dc0f1d3ecb9ba9ec7e292473d6c90b6fd201bea25181c7e66f62edf"} Nov 01 16:18:54.219452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.219420 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" event={"ID":"b50cf165-0412-4978-9947-392132261849","Type":"ContainerStarted","Data":"4856d778e1916ebbe8cf1c4d3904be475c43ff674bb4f6e906bfe1ea164a9409"} Nov 01 16:18:54.238855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.238802 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" podStartSLOduration=2.238785796 podStartE2EDuration="2.238785796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:18:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:18:54.237105614 +0000 UTC m=+1871.940271551" watchObservedRunningTime="2026-11-01 16:18:54.238785796 +0000 UTC m=+1871.941951732" Nov 01 16:18:54.373105 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:18:54.373061 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:18:54.380464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.380439 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:54.412049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.412025 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:54.517252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.508827 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72f3e0c-6521-441d-8c54-850fb767d5d8-kube-api-access-bx8nq\" (UniqueName: \"kubernetes.io/projected/f72f3e0c-6521-441d-8c54-850fb767d5d8-kube-api-access-bx8nq\") pod \"f72f3e0c-6521-441d-8c54-850fb767d5d8\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " Nov 01 16:18:54.517252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.509178 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-openshift-ca\") pod \"f72f3e0c-6521-441d-8c54-850fb767d5d8\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " Nov 01 16:18:54.517252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.509252 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-xks-ca\") pod \"f72f3e0c-6521-441d-8c54-850fb767d5d8\" (UID: \"f72f3e0c-6521-441d-8c54-850fb767d5d8\") " Nov 01 16:18:54.517252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.513616 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-openshift-ca" pod "f72f3e0c-6521-441d-8c54-850fb767d5d8" (UID: "f72f3e0c-6521-441d-8c54-850fb767d5d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:54.519618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.519586 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72f3e0c-6521-441d-8c54-850fb767d5d8-kube-api-access-bx8nq" pod "f72f3e0c-6521-441d-8c54-850fb767d5d8" (UID: "f72f3e0c-6521-441d-8c54-850fb767d5d8"). InnerVolumeSpecName "kube-api-access-bx8nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:54.528868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.528837 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-xks-ca" pod "f72f3e0c-6521-441d-8c54-850fb767d5d8" (UID: "f72f3e0c-6521-441d-8c54-850fb767d5d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:54.531535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.531510 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="751315de-016b-4107-9b93-c3d4d139b9c0" path="/var/lib/kubelet/pods/751315de-016b-4107-9b93-c3d4d139b9c0/volumes" Nov 01 16:18:54.612277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.612239 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:54.612277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.612281 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f72f3e0c-6521-441d-8c54-850fb767d5d8-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:54.612451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:54.612299 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx8nq\" (UniqueName: \"kubernetes.io/projected/f72f3e0c-6521-441d-8c54-850fb767d5d8-kube-api-access-bx8nq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:55.228898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:55.228869 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" Nov 01 16:18:55.229091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:55.228898 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88" event={"ID":"f72f3e0c-6521-441d-8c54-850fb767d5d8","Type":"ContainerDied","Data":"8e3bb3cb0eefa347cb8698d2be0093a66cb7de7b22caa92dba6cf66754f0ffbb"} Nov 01 16:18:55.229091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:55.228956 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e3bb3cb0eefa347cb8698d2be0093a66cb7de7b22caa92dba6cf66754f0ffbb" Nov 01 16:18:55.230980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:55.230953 4573 generic.go:356] "Generic (PLEG): container finished" podID="b50cf165-0412-4978-9947-392132261849" containerID="b2c8fbc46dc0f1d3ecb9ba9ec7e292473d6c90b6fd201bea25181c7e66f62edf" exitCode=0 Nov 01 16:18:55.231165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:55.231129 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" event={"ID":"b50cf165-0412-4978-9947-392132261849","Type":"ContainerDied","Data":"b2c8fbc46dc0f1d3ecb9ba9ec7e292473d6c90b6fd201bea25181c7e66f62edf"} Nov 01 16:18:55.290673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:55.290646 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:55.627523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:55.627489 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:55.729211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:55.729181 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:56.197371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.197332 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:56.513206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.513182 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:56.638505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.638473 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b50cf165-0412-4978-9947-392132261849-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50cf165-0412-4978-9947-392132261849-maas-api-xks-ca\") pod \"b50cf165-0412-4978-9947-392132261849\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " Nov 01 16:18:56.638931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.638634 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b50cf165-0412-4978-9947-392132261849-kube-api-access-7k89b\" (UniqueName: \"kubernetes.io/projected/b50cf165-0412-4978-9947-392132261849-kube-api-access-7k89b\") pod \"b50cf165-0412-4978-9947-392132261849\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " Nov 01 16:18:56.638931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.638876 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b50cf165-0412-4978-9947-392132261849-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50cf165-0412-4978-9947-392132261849-maas-api-openshift-ca\") pod \"b50cf165-0412-4978-9947-392132261849\" (UID: \"b50cf165-0412-4978-9947-392132261849\") " Nov 01 16:18:56.639275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.639254 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50cf165-0412-4978-9947-392132261849-maas-api-openshift-ca" pod "b50cf165-0412-4978-9947-392132261849" (UID: "b50cf165-0412-4978-9947-392132261849"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:18:56.639532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.639517 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b50cf165-0412-4978-9947-392132261849-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:56.640642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.640619 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50cf165-0412-4978-9947-392132261849-maas-api-xks-ca" pod "b50cf165-0412-4978-9947-392132261849" (UID: "b50cf165-0412-4978-9947-392132261849"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:18:56.640808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.640789 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50cf165-0412-4978-9947-392132261849-kube-api-access-7k89b" pod "b50cf165-0412-4978-9947-392132261849" (UID: "b50cf165-0412-4978-9947-392132261849"). InnerVolumeSpecName "kube-api-access-7k89b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:18:56.740831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.740722 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b50cf165-0412-4978-9947-392132261849-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:56.740831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.740764 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k89b\" (UniqueName: \"kubernetes.io/projected/b50cf165-0412-4978-9947-392132261849-kube-api-access-7k89b\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:18:56.906946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:56.906915 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:57.303197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:57.303167 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" Nov 01 16:18:57.303438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:57.303167 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c" event={"ID":"b50cf165-0412-4978-9947-392132261849","Type":"ContainerDied","Data":"4856d778e1916ebbe8cf1c4d3904be475c43ff674bb4f6e906bfe1ea164a9409"} Nov 01 16:18:57.303546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:57.303465 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4856d778e1916ebbe8cf1c4d3904be475c43ff674bb4f6e906bfe1ea164a9409" Nov 01 16:18:57.537792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:57.537715 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:57.638020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:57.637990 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:57.795558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:57.795494 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:58.557499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:58.557470 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:18:59.182975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:18:59.182935 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:00.018609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:00.018571 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:00.219637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:00.219603 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:00.315976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:00.315896 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:00.714002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:00.713969 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq"] Nov 01 16:19:00.714178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:00.714158 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" containerID="cri-o://58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026" gracePeriod=30 Nov 01 16:19:00.749403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:00.749365 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:00.858838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:00.858792 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:00.956648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:00.956610 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:01.536925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:01.536894 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:01.902839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:01.902735 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:02.264573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.264484 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:19:02.409403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.409373 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552"] Nov 01 16:19:02.410258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.410239 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f72f3e0c-6521-441d-8c54-850fb767d5d8" containerName="revoke-keys" Nov 01 16:19:02.410320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.410258 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72f3e0c-6521-441d-8c54-850fb767d5d8" containerName="revoke-keys" Nov 01 16:19:02.410320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.410290 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b50cf165-0412-4978-9947-392132261849" containerName="revoke-keys" Nov 01 16:19:02.410320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.410296 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b50cf165-0412-4978-9947-392132261849" containerName="revoke-keys" Nov 01 16:19:02.410414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.410400 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72f3e0c-6521-441d-8c54-850fb767d5d8" containerName="revoke-keys" Nov 01 16:19:02.410446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.410414 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b50cf165-0412-4978-9947-392132261849" containerName="revoke-keys" Nov 01 16:19:02.414599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.414576 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.418605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.418580 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:19:02.418738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.418666 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:19:02.422525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.422498 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552"] Nov 01 16:19:02.495255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.495219 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:02.602588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.602407 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.602588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.602571 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.603220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.602647 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfc7\" (UniqueName: \"kubernetes.io/projected/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-kube-api-access-dtfc7\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.704740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.704702 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.704980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.704795 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.704980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.704846 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfc7\" (UniqueName: \"kubernetes.io/projected/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-kube-api-access-dtfc7\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.705456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.705432 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.707207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.707181 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.722377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.722351 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfc7\" (UniqueName: \"kubernetes.io/projected/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-kube-api-access-dtfc7\") pod \"maas-api-revoke-sub-0934582b-pl552\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.737391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.737372 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:02.823502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.823473 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd"] Nov 01 16:19:02.823780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:02.823736 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" containerID="cri-o://5440789990a73aed36ac74c863c400e7d8b859595bc1a25ae973aebc66c8caec" gracePeriod=30 Nov 01 16:19:03.027548 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:19:03.027519 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13eeb92_2eeb_44bb_a30b_3c6843e8ffad.slice/crio-e9e017df3e433e78aea0bab7ebe653b3659eb41f015f0d2c798fe3b29c63b724 WatchSource:0}: Error finding container e9e017df3e433e78aea0bab7ebe653b3659eb41f015f0d2c798fe3b29c63b724: Status 404 returned error can't find the container with id e9e017df3e433e78aea0bab7ebe653b3659eb41f015f0d2c798fe3b29c63b724 Nov 01 16:19:03.027671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:03.027650 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552"] Nov 01 16:19:03.359235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:03.359199 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" event={"ID":"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad","Type":"ContainerStarted","Data":"e9e017df3e433e78aea0bab7ebe653b3659eb41f015f0d2c798fe3b29c63b724"} Nov 01 16:19:03.477712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:03.477677 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:03.827094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:03.827053 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb"] Nov 01 16:19:03.830429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:03.829036 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" containerID="cri-o://1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228" gracePeriod=30 Nov 01 16:19:04.340859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:04.340818 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:04.371926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:04.371895 4573 generic.go:356] "Generic (PLEG): container finished" podID="a13eeb92-2eeb-44bb-a30b-3c6843e8ffad" containerID="5251a8619777b75f54bce17cb223a66e6cd99dc513905e9410efb61af68adf21" exitCode=0 Nov 01 16:19:04.372087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:04.371978 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" event={"ID":"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad","Type":"ContainerDied","Data":"5251a8619777b75f54bce17cb223a66e6cd99dc513905e9410efb61af68adf21"} Nov 01 16:19:05.089411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.089378 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:05.572334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.572300 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:05.586983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.586951 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-xks-ca\") pod \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " Nov 01 16:19:05.588140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.587248 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-kube-api-access-dtfc7\" (UniqueName: \"kubernetes.io/projected/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-kube-api-access-dtfc7\") pod \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " Nov 01 16:19:05.588140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.587315 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-openshift-ca\") pod \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\" (UID: \"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad\") " Nov 01 16:19:05.588406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.588379 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-openshift-ca" pod "a13eeb92-2eeb-44bb-a30b-3c6843e8ffad" (UID: "a13eeb92-2eeb-44bb-a30b-3c6843e8ffad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:05.589873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.589844 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-xks-ca" pod "a13eeb92-2eeb-44bb-a30b-3c6843e8ffad" (UID: "a13eeb92-2eeb-44bb-a30b-3c6843e8ffad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:05.596220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.596193 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-kube-api-access-dtfc7" pod "a13eeb92-2eeb-44bb-a30b-3c6843e8ffad" (UID: "a13eeb92-2eeb-44bb-a30b-3c6843e8ffad"). InnerVolumeSpecName "kube-api-access-dtfc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:05.690042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.689966 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtfc7\" (UniqueName: \"kubernetes.io/projected/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-kube-api-access-dtfc7\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:05.690042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.689994 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:05.690042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.690006 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:05.852500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:05.852465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:06.052278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.052247 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:19:06.098241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098203 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-certs\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.098688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098354 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-socket\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.098688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098413 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-credential-socket\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.098688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098449 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-podinfo\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.098688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098497 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-token\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-token\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.098688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098535 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-envoy\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.098688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098540 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-certs" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.098688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098632 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istiod-ca-cert\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.098688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098684 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-kube-api-access-gnzjr\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-kube-api-access-gnzjr\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.099184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098678 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-socket" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.099184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.098739 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-data\") pod \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\" (UID: \"7238dde2-4482-4ff0-bafe-f7be1b2434c0\") " Nov 01 16:19:06.099184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.099067 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istiod-ca-cert" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:06.099334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.099301 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-data" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.099651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.099625 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-credential-socket" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.099732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.099653 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.099732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.099674 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.099732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.099688 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.099732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.099701 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.101848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.101813 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-token" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:06.101952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.101844 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-envoy" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:06.102134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.102102 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-kube-api-access-gnzjr" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "kube-api-access-gnzjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:06.103035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.103013 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-podinfo" pod "7238dde2-4482-4ff0-bafe-f7be1b2434c0" (UID: "7238dde2-4482-4ff0-bafe-f7be1b2434c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:06.201401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.201368 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.201401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.201400 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.201401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.201410 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.201695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.201419 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7238dde2-4482-4ff0-bafe-f7be1b2434c0-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.201695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.201429 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnzjr\" (UniqueName: \"kubernetes.io/projected/7238dde2-4482-4ff0-bafe-f7be1b2434c0-kube-api-access-gnzjr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:06.397562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.396580 4573 generic.go:356] "Generic (PLEG): container finished" podID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerID="58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026" exitCode=0 Nov 01 16:19:06.397562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.396668 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" Nov 01 16:19:06.397562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.396728 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" event={"ID":"7238dde2-4482-4ff0-bafe-f7be1b2434c0","Type":"ContainerDied","Data":"58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026"} Nov 01 16:19:06.397562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.396850 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq" event={"ID":"7238dde2-4482-4ff0-bafe-f7be1b2434c0","Type":"ContainerDied","Data":"79ce87922357cb95e581cb9a2f768879d18268059a740b99648fb541b88906ee"} Nov 01 16:19:06.397562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.396883 4573 scope.go:122] "RemoveContainer" containerID="58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026" Nov 01 16:19:06.400835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.400249 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" event={"ID":"a13eeb92-2eeb-44bb-a30b-3c6843e8ffad","Type":"ContainerDied","Data":"e9e017df3e433e78aea0bab7ebe653b3659eb41f015f0d2c798fe3b29c63b724"} Nov 01 16:19:06.400835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.400303 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9e017df3e433e78aea0bab7ebe653b3659eb41f015f0d2c798fe3b29c63b724" Nov 01 16:19:06.400835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.400373 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552" Nov 01 16:19:06.413863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.413838 4573 scope.go:122] "RemoveContainer" containerID="58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026" Nov 01 16:19:06.414906 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:06.414870 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026\": container with ID starting with 58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026 not found: ID does not exist" containerID="58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026" Nov 01 16:19:06.415039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.414917 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026"} err="failed to get container status \"58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026\": rpc error: code = NotFound desc = could not find container \"58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026\": container with ID starting with 58fa3f126ca66f794e86e7907673e84460c2a541e2b238a7854c6679a7ae0026 not found: ID does not exist" Nov 01 16:19:06.463343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.463308 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:06.467540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.467515 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq"] Nov 01 16:19:06.467695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.467549 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-5x7vq"] Nov 01 16:19:06.685828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:06.685694 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:07.102344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:07.102267 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8"] Nov 01 16:19:07.112321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:07.112288 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:07.112702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:07.112679 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1c65e3-tzmj8"] Nov 01 16:19:07.201022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:07.200986 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:07.573206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:07.573178 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:07.907169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:07.907134 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:08.152771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.151930 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:08.257773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.247573 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:08.357790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.357610 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf"] Nov 01 16:19:08.371651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.365097 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-375350-lncvf"] Nov 01 16:19:08.377691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.377654 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:08.463998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.446511 4573 generic.go:356] "Generic (PLEG): container finished" podID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerID="5440789990a73aed36ac74c863c400e7d8b859595bc1a25ae973aebc66c8caec" exitCode=0 Nov 01 16:19:08.463998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.446873 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" event={"ID":"42b6d892-a51a-4141-aaf2-a5a0940e43db","Type":"ContainerDied","Data":"5440789990a73aed36ac74c863c400e7d8b859595bc1a25ae973aebc66c8caec"} Nov 01 16:19:08.507507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.507475 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:19:08.508407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.508377 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" path="/var/lib/kubelet/pods/7238dde2-4482-4ff0-bafe-f7be1b2434c0/volumes" Nov 01 16:19:08.509158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.509068 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eb9defd-7a5a-442c-a649-e3cc8c4c06bc" path="/var/lib/kubelet/pods/7eb9defd-7a5a-442c-a649-e3cc8c4c06bc/volumes" Nov 01 16:19:08.510270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.509745 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd851cef-caac-4d44-bba5-956cd06852de" path="/var/lib/kubelet/pods/cd851cef-caac-4d44-bba5-956cd06852de/volumes" Nov 01 16:19:08.633280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.633219 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-kube-api-access-fv47m\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-kube-api-access-fv47m\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.633480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.633355 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-envoy\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.633480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.633470 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b6d892-a51a-4141-aaf2-a5a0940e43db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b6d892-a51a-4141-aaf2-a5a0940e43db-istiod-ca-cert\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.633603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.633551 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-data\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.633675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.633606 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-socket\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.633675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.633664 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-certs\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.633823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.633716 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-credential-socket\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.634662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.633851 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-token\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-token\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.636085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.636056 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-data" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:08.636707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.636679 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b6d892-a51a-4141-aaf2-a5a0940e43db-istiod-ca-cert" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:08.638541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.638103 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-podinfo\") pod \"42b6d892-a51a-4141-aaf2-a5a0940e43db\" (UID: \"42b6d892-a51a-4141-aaf2-a5a0940e43db\") " Nov 01 16:19:08.639496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.639467 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-credential-socket" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:08.639846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.639820 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-socket" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:08.639947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.639882 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-certs" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:08.640346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.640321 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b6d892-a51a-4141-aaf2-a5a0940e43db-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.640437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.640352 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.640437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.640370 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.640437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.640384 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.640437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.640397 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.641919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.641884 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-podinfo" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:08.643973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.643937 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-kube-api-access-fv47m" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "kube-api-access-fv47m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:08.647537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.647499 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-envoy" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:08.660794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.660490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:08.660794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.660615 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-token" pod "42b6d892-a51a-4141-aaf2-a5a0940e43db" (UID: "42b6d892-a51a-4141-aaf2-a5a0940e43db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:08.741887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.741799 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv47m\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-kube-api-access-fv47m\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.741887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.741834 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.741887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.741851 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:08.741887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:08.741865 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b6d892-a51a-4141-aaf2-a5a0940e43db-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.113185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.113155 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 16:19:09.249864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.249821 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-envoy\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.250576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.250024 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-socket\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.250576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.250089 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istiod-ca-cert\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.250576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.250155 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-kube-api-access-lf4jf\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-kube-api-access-lf4jf\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.250576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.250225 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-token\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-token\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.250818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.250649 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-podinfo\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.250818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.250703 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-certs\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.250936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.250831 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-data\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.250936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.250894 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-credential-socket\") pod \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\" (UID: \"1af93991-fb33-4054-82cd-5e76ebfa4cbb\") " Nov 01 16:19:09.255054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.255026 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-socket" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:09.255401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.255096 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-certs" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:09.255558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.255411 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-data" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:09.255728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.255706 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istiod-ca-cert" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:09.255917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.255895 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-credential-socket" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:09.259140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.258151 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-token" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:09.259140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.258884 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-envoy" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:09.259140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.258958 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-kube-api-access-lf4jf" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "kube-api-access-lf4jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:09.261377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.261328 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-podinfo" pod "1af93991-fb33-4054-82cd-5e76ebfa4cbb" (UID: "1af93991-fb33-4054-82cd-5e76ebfa4cbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:09.317922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.317883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:09.354344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354315 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.354344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354344 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf4jf\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-kube-api-access-lf4jf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.354522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354355 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.354522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354365 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.354522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354373 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.354522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354382 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.354522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354391 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.354522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354399 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.354522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.354407 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af93991-fb33-4054-82cd-5e76ebfa4cbb-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:09.459835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.459790 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" event={"ID":"42b6d892-a51a-4141-aaf2-a5a0940e43db","Type":"ContainerDied","Data":"69b49e976241549fae73952b446d379290a19804a323f81606b87e441b105a0c"} Nov 01 16:19:09.460025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.459841 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd" Nov 01 16:19:09.460025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.459872 4573 scope.go:122] "RemoveContainer" containerID="5440789990a73aed36ac74c863c400e7d8b859595bc1a25ae973aebc66c8caec" Nov 01 16:19:09.465068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.465042 4573 generic.go:356] "Generic (PLEG): container finished" podID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerID="1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228" exitCode=0 Nov 01 16:19:09.465207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.465186 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" event={"ID":"1af93991-fb33-4054-82cd-5e76ebfa4cbb","Type":"ContainerDied","Data":"1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228"} Nov 01 16:19:09.465295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.465230 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" event={"ID":"1af93991-fb33-4054-82cd-5e76ebfa4cbb","Type":"ContainerDied","Data":"08c3885808177258ccf876760931f67c018ea2d65d903f3c6c7bc1760604b26c"} Nov 01 16:19:09.465295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.465238 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb" Nov 01 16:19:09.478470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.478446 4573 scope.go:122] "RemoveContainer" containerID="1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228" Nov 01 16:19:09.500612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.500581 4573 scope.go:122] "RemoveContainer" containerID="1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228" Nov 01 16:19:09.506983 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:09.501404 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228\": container with ID starting with 1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228 not found: ID does not exist" containerID="1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228" Nov 01 16:19:09.506983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.501984 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228"} err="failed to get container status \"1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228\": rpc error: code = NotFound desc = could not find container \"1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228\": container with ID starting with 1226316d21f87507ead9b768b93bb5c3f4f88b1f7a85c676ff7be81624f9c228 not found: ID does not exist" Nov 01 16:19:09.506983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.502417 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd"] Nov 01 16:19:09.524631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.524552 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-rj7xd"] Nov 01 16:19:09.534121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.534089 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:09.547099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.547072 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb"] Nov 01 16:19:09.551426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.551401 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-klsmb"] Nov 01 16:19:09.997692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.997654 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5"] Nov 01 16:19:09.998493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998476 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" Nov 01 16:19:09.998557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998494 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" Nov 01 16:19:09.998557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998507 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" Nov 01 16:19:09.998557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998512 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" Nov 01 16:19:09.998557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998525 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a13eeb92-2eeb-44bb-a30b-3c6843e8ffad" containerName="revoke-keys" Nov 01 16:19:09.998557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998530 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13eeb92-2eeb-44bb-a30b-3c6843e8ffad" containerName="revoke-keys" Nov 01 16:19:09.998714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998572 4573 cpu_manager.go:414] "containerMap: removing container" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" Nov 01 16:19:09.998714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998577 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" Nov 01 16:19:09.998714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998686 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7238dde2-4482-4ff0-bafe-f7be1b2434c0" containerName="istio-proxy" Nov 01 16:19:09.998714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998697 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" containerName="istio-proxy" Nov 01 16:19:09.998714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998705 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13eeb92-2eeb-44bb-a30b-3c6843e8ffad" containerName="revoke-keys" Nov 01 16:19:09.998900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:09.998718 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" containerName="istio-proxy" Nov 01 16:19:10.004602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.004574 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.007796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.007729 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:19:10.008248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.008227 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:19:10.010899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.010877 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5"] Nov 01 16:19:10.176033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.175995 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hns8\" (UniqueName: \"kubernetes.io/projected/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-kube-api-access-4hns8\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.176219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.176176 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.176330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.176307 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.259240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.259154 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:10.278042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.278018 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.278201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.278102 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.278272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.278258 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hns8\" (UniqueName: \"kubernetes.io/projected/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-kube-api-access-4hns8\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.278825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.278767 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.280544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.280523 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.296540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.296515 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hns8\" (UniqueName: \"kubernetes.io/projected/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-kube-api-access-4hns8\") pod \"maas-api-revoke-sub-c6590b71-82xz5\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.336129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.336049 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:10.463954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.463927 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af93991-fb33-4054-82cd-5e76ebfa4cbb" path="/var/lib/kubelet/pods/1af93991-fb33-4054-82cd-5e76ebfa4cbb/volumes" Nov 01 16:19:10.464510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.464495 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b6d892-a51a-4141-aaf2-a5a0940e43db" path="/var/lib/kubelet/pods/42b6d892-a51a-4141-aaf2-a5a0940e43db/volumes" Nov 01 16:19:10.654006 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:19:10.653975 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdcd16e_5143_4a27_8e53_c11f75e1e99a.slice/crio-0914abba1677c71b45f3e9f2363fccc1e519f1fe8c3b448d8ba5e4c0ec2d6637 WatchSource:0}: Error finding container 0914abba1677c71b45f3e9f2363fccc1e519f1fe8c3b448d8ba5e4c0ec2d6637: Status 404 returned error can't find the container with id 0914abba1677c71b45f3e9f2363fccc1e519f1fe8c3b448d8ba5e4c0ec2d6637 Nov 01 16:19:10.654230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:10.654210 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5"] Nov 01 16:19:11.054745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:11.054711 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:11.486714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:11.486676 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" event={"ID":"6fdcd16e-5143-4a27-8e53-c11f75e1e99a","Type":"ContainerStarted","Data":"92e58b37d7243fede254ca6162349a182d0d5eefb048681b3aac14060ccd5c45"} Nov 01 16:19:11.487127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:11.486737 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" event={"ID":"6fdcd16e-5143-4a27-8e53-c11f75e1e99a","Type":"ContainerStarted","Data":"0914abba1677c71b45f3e9f2363fccc1e519f1fe8c3b448d8ba5e4c0ec2d6637"} Nov 01 16:19:11.506597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:11.506547 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" podStartSLOduration=2.506534784 podStartE2EDuration="2.506534784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:19:11.50135124 +0000 UTC m=+1889.204517177" watchObservedRunningTime="2026-11-01 16:19:11.506534784 +0000 UTC m=+1889.209700721" Nov 01 16:19:11.770739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:11.770708 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:12.496321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:12.496291 4573 generic.go:356] "Generic (PLEG): container finished" podID="6fdcd16e-5143-4a27-8e53-c11f75e1e99a" containerID="92e58b37d7243fede254ca6162349a182d0d5eefb048681b3aac14060ccd5c45" exitCode=0 Nov 01 16:19:12.496723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:12.496379 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" event={"ID":"6fdcd16e-5143-4a27-8e53-c11f75e1e99a","Type":"ContainerDied","Data":"92e58b37d7243fede254ca6162349a182d0d5eefb048681b3aac14060ccd5c45"} Nov 01 16:19:12.660236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:12.657887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:13.391774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.384243 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:13.836008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.835979 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:13.971877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.971796 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-kube-api-access-4hns8\" (UniqueName: \"kubernetes.io/projected/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-kube-api-access-4hns8\") pod \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " Nov 01 16:19:13.972102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.972085 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-xks-ca\") pod \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " Nov 01 16:19:13.972200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.972144 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-openshift-ca\") pod \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\" (UID: \"6fdcd16e-5143-4a27-8e53-c11f75e1e99a\") " Nov 01 16:19:13.972602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.972572 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-openshift-ca" pod "6fdcd16e-5143-4a27-8e53-c11f75e1e99a" (UID: "6fdcd16e-5143-4a27-8e53-c11f75e1e99a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:13.972840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.972813 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:13.974221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.974196 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-kube-api-access-4hns8" pod "6fdcd16e-5143-4a27-8e53-c11f75e1e99a" (UID: "6fdcd16e-5143-4a27-8e53-c11f75e1e99a"). InnerVolumeSpecName "kube-api-access-4hns8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:13.974296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.974269 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-xks-ca" pod "6fdcd16e-5143-4a27-8e53-c11f75e1e99a" (UID: "6fdcd16e-5143-4a27-8e53-c11f75e1e99a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:13.999589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:13.999518 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:14.075899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.075595 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:14.075899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.075637 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hns8\" (UniqueName: \"kubernetes.io/projected/6fdcd16e-5143-4a27-8e53-c11f75e1e99a-kube-api-access-4hns8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:14.101622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.101587 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:14.312426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.312335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:14.521591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.521556 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:14.664780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.656005 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" event={"ID":"6fdcd16e-5143-4a27-8e53-c11f75e1e99a","Type":"ContainerDied","Data":"0914abba1677c71b45f3e9f2363fccc1e519f1fe8c3b448d8ba5e4c0ec2d6637"} Nov 01 16:19:14.664780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.656090 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0914abba1677c71b45f3e9f2363fccc1e519f1fe8c3b448d8ba5e4c0ec2d6637" Nov 01 16:19:14.669003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.665169 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5" Nov 01 16:19:14.820763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.820720 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9"] Nov 01 16:19:14.821570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.821547 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6fdcd16e-5143-4a27-8e53-c11f75e1e99a" containerName="revoke-keys" Nov 01 16:19:14.821570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.821569 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdcd16e-5143-4a27-8e53-c11f75e1e99a" containerName="revoke-keys" Nov 01 16:19:14.821820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.821707 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdcd16e-5143-4a27-8e53-c11f75e1e99a" containerName="revoke-keys" Nov 01 16:19:14.826216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.826193 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:14.829953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.829930 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:19:14.831854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.830103 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:19:14.833165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.833140 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9"] Nov 01 16:19:14.933340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.933306 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:14.933698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.933351 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:14.933698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:14.933445 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtzdw\" (UniqueName: \"kubernetes.io/projected/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-kube-api-access-wtzdw\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:15.034957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.034911 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:15.035113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.035004 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtzdw\" (UniqueName: \"kubernetes.io/projected/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-kube-api-access-wtzdw\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:15.035355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.035339 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:15.035970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.035950 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:15.037304 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.037285 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:15.042937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.042916 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtzdw\" (UniqueName: \"kubernetes.io/projected/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-kube-api-access-wtzdw\") pod \"maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:15.151088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.151061 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:15.367727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.367693 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:15.489199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.489164 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9"] Nov 01 16:19:15.676341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.676121 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" event={"ID":"10fe679f-38da-4dc3-81bf-4b4e7f92adbe","Type":"ContainerStarted","Data":"90998e2b42efa423ceba16db684bf2db2fd2d97a4b43d94e6b43594bc1e4e99b"} Nov 01 16:19:15.889360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.889307 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:15.994999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:15.994938 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:16.165421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:16.165335 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk"] Nov 01 16:19:16.165633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:16.165606 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" containerID="cri-o://ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b" gracePeriod=30 Nov 01 16:19:16.176672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:16.176647 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:16.291982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:16.291952 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:16.687624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:16.687586 4573 generic.go:356] "Generic (PLEG): container finished" podID="10fe679f-38da-4dc3-81bf-4b4e7f92adbe" containerID="4e7b2f0c9f98b2a9898cec9e6d6172a9eb8f7c3daef2c67c559f0e08bc011215" exitCode=0 Nov 01 16:19:16.687775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:16.687630 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" event={"ID":"10fe679f-38da-4dc3-81bf-4b4e7f92adbe","Type":"ContainerDied","Data":"4e7b2f0c9f98b2a9898cec9e6d6172a9eb8f7c3daef2c67c559f0e08bc011215"} Nov 01 16:19:17.179374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:17.179339 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:17.878859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:17.878833 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:17.887682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:17.887659 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:17.997695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:17.997596 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-kube-api-access-wtzdw\" (UniqueName: \"kubernetes.io/projected/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-kube-api-access-wtzdw\") pod \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " Nov 01 16:19:17.998486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:17.997952 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-openshift-ca\") pod \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " Nov 01 16:19:17.998486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:17.998040 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-xks-ca\") pod \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\" (UID: \"10fe679f-38da-4dc3-81bf-4b4e7f92adbe\") " Nov 01 16:19:17.998486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:17.998445 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-openshift-ca" pod "10fe679f-38da-4dc3-81bf-4b4e7f92adbe" (UID: "10fe679f-38da-4dc3-81bf-4b4e7f92adbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:17.998899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:17.998870 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:18.000696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:18.000662 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-kube-api-access-wtzdw" pod "10fe679f-38da-4dc3-81bf-4b4e7f92adbe" (UID: "10fe679f-38da-4dc3-81bf-4b4e7f92adbe"). InnerVolumeSpecName "kube-api-access-wtzdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:18.000842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:18.000808 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-xks-ca" pod "10fe679f-38da-4dc3-81bf-4b4e7f92adbe" (UID: "10fe679f-38da-4dc3-81bf-4b4e7f92adbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:18.105096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:18.105059 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:18.105096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:18.105096 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtzdw\" (UniqueName: \"kubernetes.io/projected/10fe679f-38da-4dc3-81bf-4b4e7f92adbe-kube-api-access-wtzdw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:18.707048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:18.707007 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:18.720658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:18.720558 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" event={"ID":"10fe679f-38da-4dc3-81bf-4b4e7f92adbe","Type":"ContainerDied","Data":"90998e2b42efa423ceba16db684bf2db2fd2d97a4b43d94e6b43594bc1e4e99b"} Nov 01 16:19:18.720658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:18.720607 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90998e2b42efa423ceba16db684bf2db2fd2d97a4b43d94e6b43594bc1e4e99b" Nov 01 16:19:18.720658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:18.720662 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9" Nov 01 16:19:19.326278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:19.326242 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:20.256076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.256042 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:20.381698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.381663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:20.651020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.650984 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7"] Nov 01 16:19:20.651896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.651879 4573 cpu_manager.go:414] "containerMap: removing container" podUID="10fe679f-38da-4dc3-81bf-4b4e7f92adbe" containerName="revoke-keys" Nov 01 16:19:20.651959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.651898 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10fe679f-38da-4dc3-81bf-4b4e7f92adbe" containerName="revoke-keys" Nov 01 16:19:20.652065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.652055 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="10fe679f-38da-4dc3-81bf-4b4e7f92adbe" containerName="revoke-keys" Nov 01 16:19:20.656251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.656223 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.662454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.662425 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:19:20.663152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.663128 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:19:20.674838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.671714 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7"] Nov 01 16:19:20.703386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.703352 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s82f\" (UniqueName: \"kubernetes.io/projected/7f7ab1be-f802-4541-8416-4bf3085b07ae-kube-api-access-2s82f\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.703550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.703413 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.703654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.703630 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.804816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.804740 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.805009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.804916 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.805115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.805099 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s82f\" (UniqueName: \"kubernetes.io/projected/7f7ab1be-f802-4541-8416-4bf3085b07ae-kube-api-access-2s82f\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.805327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.805303 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.807426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.807405 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.815958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.815932 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s82f\" (UniqueName: \"kubernetes.io/projected/7f7ab1be-f802-4541-8416-4bf3085b07ae-kube-api-access-2s82f\") pod \"maas-api-revoke-keys-e2e-inf-a-784444-s47d7\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:20.980362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:20.980282 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:21.266345 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:19:21.266315 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7ab1be_f802_4541_8416_4bf3085b07ae.slice/crio-7e8eda67ba5ae8beb5ae20b45ff7c7a8f9e8febfef4cc73e0731a4629d5057c1 WatchSource:0}: Error finding container 7e8eda67ba5ae8beb5ae20b45ff7c7a8f9e8febfef4cc73e0731a4629d5057c1: Status 404 returned error can't find the container with id 7e8eda67ba5ae8beb5ae20b45ff7c7a8f9e8febfef4cc73e0731a4629d5057c1 Nov 01 16:19:21.267244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.267221 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7"] Nov 01 16:19:21.286035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.286009 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:21.514331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.514260 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:19:21.628953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.628894 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-token\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-token\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.628953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.628941 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-certs\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.629217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629000 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istiod-ca-cert\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.629217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629084 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-socket\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.629217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629128 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-envoy\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.629402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629216 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-credential-socket\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.629402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629313 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-data\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.629402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629337 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-certs" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.629882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629407 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-podinfo\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.630027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629917 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-kube-api-access-c9bbw\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-kube-api-access-c9bbw\") pod \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\" (UID: \"fa97b19f-b620-4e57-ab66-1abe674d0b0e\") " Nov 01 16:19:21.630027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629460 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-socket" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.630027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629774 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istiod-ca-cert" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:21.630027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629736 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-data" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.630027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.629826 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-credential-socket" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.631129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.631028 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.631129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.631060 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.631129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.631077 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.631129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.631093 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.631129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.631108 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.632103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.632054 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-podinfo" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:21.632838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.632802 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-kube-api-access-c9bbw" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "kube-api-access-c9bbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.633108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.633069 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-envoy" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:21.633224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.633205 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-token" pod "fa97b19f-b620-4e57-ab66-1abe674d0b0e" (UID: "fa97b19f-b620-4e57-ab66-1abe674d0b0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:21.733030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.732996 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.733030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.733026 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bbw\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-kube-api-access-c9bbw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.733030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.733038 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.733361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.733048 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa97b19f-b620-4e57-ab66-1abe674d0b0e-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:21.755411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.755368 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" event={"ID":"7f7ab1be-f802-4541-8416-4bf3085b07ae","Type":"ContainerStarted","Data":"7e8eda67ba5ae8beb5ae20b45ff7c7a8f9e8febfef4cc73e0731a4629d5057c1"} Nov 01 16:19:21.757171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.757134 4573 generic.go:356] "Generic (PLEG): container finished" podID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerID="ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b" exitCode=0 Nov 01 16:19:21.757334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.757183 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" Nov 01 16:19:21.757334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.757306 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" event={"ID":"fa97b19f-b620-4e57-ab66-1abe674d0b0e","Type":"ContainerDied","Data":"ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b"} Nov 01 16:19:21.757468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.757349 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk" event={"ID":"fa97b19f-b620-4e57-ab66-1abe674d0b0e","Type":"ContainerDied","Data":"33017eab463c2ca1514b5caab4c58e9ef14406f51ec6c6cb931a0843acc23a35"} Nov 01 16:19:21.757468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.757378 4573 scope.go:122] "RemoveContainer" containerID="ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b" Nov 01 16:19:21.772219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.772155 4573 scope.go:122] "RemoveContainer" containerID="ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b" Nov 01 16:19:21.772490 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:21.772463 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b\": container with ID starting with ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b not found: ID does not exist" containerID="ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b" Nov 01 16:19:21.772644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.772503 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b"} err="failed to get container status \"ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b\": rpc error: code = NotFound desc = could not find container \"ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b\": container with ID starting with ccc50056f5ed932ef45a27931cca44fde655038b4bc6aaca8ad78652fcf8a46b not found: ID does not exist" Nov 01 16:19:21.799373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.799346 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk"] Nov 01 16:19:21.803966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:21.803942 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-375350-openshift-default-5c64667db7-dr7lk"] Nov 01 16:19:22.026497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:22.026415 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:22.461381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:22.461345 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" path="/var/lib/kubelet/pods/fa97b19f-b620-4e57-ab66-1abe674d0b0e/volumes" Nov 01 16:19:22.667903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:22.667867 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:22.802910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:22.802822 4573 generic.go:356] "Generic (PLEG): container finished" podID="7f7ab1be-f802-4541-8416-4bf3085b07ae" containerID="a4067573b31634aa0164e760935a24ff0bf5ae2cbfb519062f0baaf2d585b339" exitCode=0 Nov 01 16:19:22.802910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:22.802865 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" event={"ID":"7f7ab1be-f802-4541-8416-4bf3085b07ae","Type":"ContainerDied","Data":"a4067573b31634aa0164e760935a24ff0bf5ae2cbfb519062f0baaf2d585b339"} Nov 01 16:19:23.288681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:23.288642 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:23.670541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:23.670479 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:24.011540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.011516 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:24.065450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.065414 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-openshift-ca\") pod \"7f7ab1be-f802-4541-8416-4bf3085b07ae\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " Nov 01 16:19:24.065647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.065590 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-xks-ca\") pod \"7f7ab1be-f802-4541-8416-4bf3085b07ae\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " Nov 01 16:19:24.065647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.065638 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7ab1be-f802-4541-8416-4bf3085b07ae-kube-api-access-2s82f\" (UniqueName: \"kubernetes.io/projected/7f7ab1be-f802-4541-8416-4bf3085b07ae-kube-api-access-2s82f\") pod \"7f7ab1be-f802-4541-8416-4bf3085b07ae\" (UID: \"7f7ab1be-f802-4541-8416-4bf3085b07ae\") " Nov 01 16:19:24.065904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.065879 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-openshift-ca" pod "7f7ab1be-f802-4541-8416-4bf3085b07ae" (UID: "7f7ab1be-f802-4541-8416-4bf3085b07ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:24.068202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.068169 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.081083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.069100 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7ab1be-f802-4541-8416-4bf3085b07ae-kube-api-access-2s82f" pod "7f7ab1be-f802-4541-8416-4bf3085b07ae" (UID: "7f7ab1be-f802-4541-8416-4bf3085b07ae"). InnerVolumeSpecName "kube-api-access-2s82f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:24.086605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.086581 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-xks-ca" pod "7f7ab1be-f802-4541-8416-4bf3085b07ae" (UID: "7f7ab1be-f802-4541-8416-4bf3085b07ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:24.156531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.156498 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:24.169828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.169800 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f7ab1be-f802-4541-8416-4bf3085b07ae-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.169828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.169828 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s82f\" (UniqueName: \"kubernetes.io/projected/7f7ab1be-f802-4541-8416-4bf3085b07ae-kube-api-access-2s82f\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:24.440822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.440785 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:24.459095 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:24.459049 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:19:24.468485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.468448 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c"] Nov 01 16:19:24.469337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.469313 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7f7ab1be-f802-4541-8416-4bf3085b07ae" containerName="revoke-keys" Nov 01 16:19:24.469337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.469335 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7ab1be-f802-4541-8416-4bf3085b07ae" containerName="revoke-keys" Nov 01 16:19:24.469539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.469368 4573 cpu_manager.go:414] "containerMap: removing container" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" Nov 01 16:19:24.469539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.469374 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" Nov 01 16:19:24.469539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.469514 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7ab1be-f802-4541-8416-4bf3085b07ae" containerName="revoke-keys" Nov 01 16:19:24.469539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.469531 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa97b19f-b620-4e57-ab66-1abe674d0b0e" containerName="istio-proxy" Nov 01 16:19:24.475193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.475169 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c"] Nov 01 16:19:24.475316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.475270 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.550542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.550509 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:24.645419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.645386 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5m7d\" (UniqueName: \"kubernetes.io/projected/6ac4c202-9491-4aa1-a04c-fe74655273a6-kube-api-access-q5m7d\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.645597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.645458 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.645597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.645528 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.747284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.747202 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.747284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.747244 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.747898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.747499 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5m7d\" (UniqueName: \"kubernetes.io/projected/6ac4c202-9491-4aa1-a04c-fe74655273a6-kube-api-access-q5m7d\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.748078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.748052 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.749845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.749827 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.755858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.755837 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5m7d\" (UniqueName: \"kubernetes.io/projected/6ac4c202-9491-4aa1-a04c-fe74655273a6-kube-api-access-q5m7d\") pod \"maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.799745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.799721 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:24.823448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.823422 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" Nov 01 16:19:24.823586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.823446 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7" event={"ID":"7f7ab1be-f802-4541-8416-4bf3085b07ae","Type":"ContainerDied","Data":"7e8eda67ba5ae8beb5ae20b45ff7c7a8f9e8febfef4cc73e0731a4629d5057c1"} Nov 01 16:19:24.823586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.823498 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e8eda67ba5ae8beb5ae20b45ff7c7a8f9e8febfef4cc73e0731a4629d5057c1" Nov 01 16:19:24.865098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:24.865069 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:25.084515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.084432 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:25.100846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.100810 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c"] Nov 01 16:19:25.101088 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:19:25.101064 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac4c202_9491_4aa1_a04c_fe74655273a6.slice/crio-66333cad40d6a5678d2a34b22e28f792169aaefdab449e6584b45de41eade3d2 WatchSource:0}: Error finding container 66333cad40d6a5678d2a34b22e28f792169aaefdab449e6584b45de41eade3d2: Status 404 returned error can't find the container with id 66333cad40d6a5678d2a34b22e28f792169aaefdab449e6584b45de41eade3d2 Nov 01 16:19:25.184934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.184910 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:25.540775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.538664 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:25.635428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.635398 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:25.833409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.833323 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" event={"ID":"6ac4c202-9491-4aa1-a04c-fe74655273a6","Type":"ContainerStarted","Data":"659ebf5de29d44d7736d4d57be52f8c38a2f52e4d4f32e00fd82610faf0767e7"} Nov 01 16:19:25.833409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.833382 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" event={"ID":"6ac4c202-9491-4aa1-a04c-fe74655273a6","Type":"ContainerStarted","Data":"66333cad40d6a5678d2a34b22e28f792169aaefdab449e6584b45de41eade3d2"} Nov 01 16:19:25.850600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.850548 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" podStartSLOduration=1.850535162 podStartE2EDuration="1.850535162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:19:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:19:25.848399227 +0000 UTC m=+1903.551565160" watchObservedRunningTime="2026-11-01 16:19:25.850535162 +0000 UTC m=+1903.553701098" Nov 01 16:19:25.935113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:25.935080 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:26.843949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:26.843906 4573 generic.go:356] "Generic (PLEG): container finished" podID="6ac4c202-9491-4aa1-a04c-fe74655273a6" containerID="659ebf5de29d44d7736d4d57be52f8c38a2f52e4d4f32e00fd82610faf0767e7" exitCode=0 Nov 01 16:19:26.843949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:26.843952 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" event={"ID":"6ac4c202-9491-4aa1-a04c-fe74655273a6","Type":"ContainerDied","Data":"659ebf5de29d44d7736d4d57be52f8c38a2f52e4d4f32e00fd82610faf0767e7"} Nov 01 16:19:26.864795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:26.864683 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:27.648682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:27.648640 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:28.038242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.038215 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:28.133959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.133928 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-openshift-ca\") pod \"6ac4c202-9491-4aa1-a04c-fe74655273a6\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " Nov 01 16:19:28.134118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.134000 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-xks-ca\") pod \"6ac4c202-9491-4aa1-a04c-fe74655273a6\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " Nov 01 16:19:28.134118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.134105 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac4c202-9491-4aa1-a04c-fe74655273a6-kube-api-access-q5m7d\" (UniqueName: \"kubernetes.io/projected/6ac4c202-9491-4aa1-a04c-fe74655273a6-kube-api-access-q5m7d\") pod \"6ac4c202-9491-4aa1-a04c-fe74655273a6\" (UID: \"6ac4c202-9491-4aa1-a04c-fe74655273a6\") " Nov 01 16:19:28.134359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.134337 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-openshift-ca" pod "6ac4c202-9491-4aa1-a04c-fe74655273a6" (UID: "6ac4c202-9491-4aa1-a04c-fe74655273a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:28.134819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.134798 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:28.136151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.136128 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-xks-ca" pod "6ac4c202-9491-4aa1-a04c-fe74655273a6" (UID: "6ac4c202-9491-4aa1-a04c-fe74655273a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:28.136277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.136259 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac4c202-9491-4aa1-a04c-fe74655273a6-kube-api-access-q5m7d" pod "6ac4c202-9491-4aa1-a04c-fe74655273a6" (UID: "6ac4c202-9491-4aa1-a04c-fe74655273a6"). InnerVolumeSpecName "kube-api-access-q5m7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:28.236135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.236060 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac4c202-9491-4aa1-a04c-fe74655273a6-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:28.236135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.236088 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5m7d\" (UniqueName: \"kubernetes.io/projected/6ac4c202-9491-4aa1-a04c-fe74655273a6-kube-api-access-q5m7d\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:28.387115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.387081 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:28.863009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.862980 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" Nov 01 16:19:28.863209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.863017 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c" event={"ID":"6ac4c202-9491-4aa1-a04c-fe74655273a6","Type":"ContainerDied","Data":"66333cad40d6a5678d2a34b22e28f792169aaefdab449e6584b45de41eade3d2"} Nov 01 16:19:28.863209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:28.863067 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66333cad40d6a5678d2a34b22e28f792169aaefdab449e6584b45de41eade3d2" Nov 01 16:19:29.081427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:29.081394 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:29.925917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:29.925886 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:30.655524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:30.655484 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:30.865121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:30.865083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:30.959442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:30.959371 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:31.135772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:31.123204 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq"] Nov 01 16:19:31.135772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:31.123450 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" containerID="cri-o://55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b" gracePeriod=30 Nov 01 16:19:31.572957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:31.572917 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:31.728480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:31.728444 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:32.367571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:32.367533 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:33.156459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:33.156412 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:34.219028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:34.202627 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:34.546057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:34.545986 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g"] Nov 01 16:19:34.559646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:34.559619 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2"] Nov 01 16:19:34.621444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:34.611844 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:34.708970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:34.708936 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:35.045592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:35.045532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:35.852010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:35.851975 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:36.134394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.134363 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:19:36.134593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.134431 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:19:36.459137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.459102 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:36.591442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.591410 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 16:19:36.617888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.617852 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:36.709141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.709048 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:36.717341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.717312 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7"] Nov 01 16:19:36.721936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.721910 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-784444-s47d7"] Nov 01 16:19:36.788664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.788628 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-credential-socket\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.788863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.788720 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-socket\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.789048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.789016 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-credential-socket" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:36.789426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.789399 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c25e01d-b4b7-4198-a519-d1c6614e8893-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c25e01d-b4b7-4198-a519-d1c6614e8893-istiod-ca-cert\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.789541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.789461 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-data\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.790044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.789598 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-kube-api-access-s7wsm\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-kube-api-access-s7wsm\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.790044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.789774 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-podinfo\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.790044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.789814 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-token\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-token\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.790044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.789886 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-envoy\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.790044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.789957 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-certs\") pod \"9c25e01d-b4b7-4198-a519-d1c6614e8893\" (UID: \"9c25e01d-b4b7-4198-a519-d1c6614e8893\") " Nov 01 16:19:36.790044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.790026 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c25e01d-b4b7-4198-a519-d1c6614e8893-istiod-ca-cert" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:36.790411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.790253 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-socket" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:36.790799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.790730 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.790799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.790767 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.790799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.790783 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c25e01d-b4b7-4198-a519-d1c6614e8893-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.791029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.791009 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-data" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:36.791029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.791020 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-certs" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:36.792946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.792919 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-podinfo" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:36.793270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.793244 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-token" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:36.795236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.795210 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-kube-api-access-s7wsm" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "kube-api-access-s7wsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:36.795332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.795239 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-envoy" pod "9c25e01d-b4b7-4198-a519-d1c6614e8893" (UID: "9c25e01d-b4b7-4198-a519-d1c6614e8893"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:36.822378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.822352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:36.892689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.892656 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.892689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.892687 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.892689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.892697 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.893144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.892707 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.893144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.892717 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c25e01d-b4b7-4198-a519-d1c6614e8893-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.893144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.892725 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wsm\" (UniqueName: \"kubernetes.io/projected/9c25e01d-b4b7-4198-a519-d1c6614e8893-kube-api-access-s7wsm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:36.933486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:36.933452 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:37.004417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.004271 4573 generic.go:356] "Generic (PLEG): container finished" podID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerID="55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b" exitCode=0 Nov 01 16:19:37.004655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.004314 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" Nov 01 16:19:37.004848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.004346 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" event={"ID":"9c25e01d-b4b7-4198-a519-d1c6614e8893","Type":"ContainerDied","Data":"55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b"} Nov 01 16:19:37.004978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.004889 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq" event={"ID":"9c25e01d-b4b7-4198-a519-d1c6614e8893","Type":"ContainerDied","Data":"ef15d2f298d5d4fa4089150c7f818a2c8c2776da5e5a492fab71cfa04e4204b3"} Nov 01 16:19:37.004978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.004928 4573 scope.go:122] "RemoveContainer" containerID="55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b" Nov 01 16:19:37.007571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.007549 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9"] Nov 01 16:19:37.013199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.013087 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2ce08a-nz7z9"] Nov 01 16:19:37.020984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.020965 4573 scope.go:122] "RemoveContainer" containerID="55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b" Nov 01 16:19:37.021266 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:37.021247 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b\": container with ID starting with 55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b not found: ID does not exist" containerID="55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b" Nov 01 16:19:37.021321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.021274 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b"} err="failed to get container status \"55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b\": rpc error: code = NotFound desc = could not find container \"55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b\": container with ID starting with 55d9dc8de9fa9662222e968d41c9ea6882ae9e694decaf2be7bc5f1a9081023b not found: ID does not exist" Nov 01 16:19:37.051118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.051091 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq"] Nov 01 16:19:37.077776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.074084 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-ddbpq"] Nov 01 16:19:37.176775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.175996 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:37.384038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.383992 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:37.464160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.464128 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:37.473783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.473743 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c"] Nov 01 16:19:37.478028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.477999 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2001b0-lz85c"] Nov 01 16:19:37.551123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.551077 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:37.933777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:37.933723 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:38.361419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:38.361392 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:38.460299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:38.460267 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10fe679f-38da-4dc3-81bf-4b4e7f92adbe" path="/var/lib/kubelet/pods/10fe679f-38da-4dc3-81bf-4b4e7f92adbe/volumes" Nov 01 16:19:38.460832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:38.460816 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac4c202-9491-4aa1-a04c-fe74655273a6" path="/var/lib/kubelet/pods/6ac4c202-9491-4aa1-a04c-fe74655273a6/volumes" Nov 01 16:19:38.461430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:38.461410 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7ab1be-f802-4541-8416-4bf3085b07ae" path="/var/lib/kubelet/pods/7f7ab1be-f802-4541-8416-4bf3085b07ae/volumes" Nov 01 16:19:38.462206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:38.462190 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" path="/var/lib/kubelet/pods/9c25e01d-b4b7-4198-a519-d1c6614e8893/volumes" Nov 01 16:19:38.994129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:38.994076 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:39.406572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:39.406542 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:39.558604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:39.558558 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" podUID="73f2b95e-884c-4b6c-8a93-4b47fccc47de" containerName="payload-processing" containerID="cri-o://e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2" gracePeriod=25 Nov 01 16:19:39.572315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:39.572058 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" podUID="b364e1c7-df3f-4daa-b582-d11057bd4585" containerName="payload-pre-processing" containerID="cri-o://33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e" gracePeriod=25 Nov 01 16:19:40.079458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:40.079407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:40.847932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:40.847901 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:41.461648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:41.461614 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:42.268002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:42.267968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:43.200214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:43.200177 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:44.357787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:44.357733 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:44.925914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:44.925882 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:45.028658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.028624 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:45.143714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.143675 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:45.259601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.259485 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:45.312477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.312441 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9"] Nov 01 16:19:45.312862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.312830 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" containerID="cri-o://4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae" gracePeriod=30 Nov 01 16:19:45.356063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.356029 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:45.717139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.717098 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb"] Nov 01 16:19:45.718112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.718091 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6ac4c202-9491-4aa1-a04c-fe74655273a6" containerName="revoke-keys" Nov 01 16:19:45.718112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.718111 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac4c202-9491-4aa1-a04c-fe74655273a6" containerName="revoke-keys" Nov 01 16:19:45.718295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.718140 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" Nov 01 16:19:45.718295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.718146 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" Nov 01 16:19:45.718409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.718349 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac4c202-9491-4aa1-a04c-fe74655273a6" containerName="revoke-keys" Nov 01 16:19:45.718409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.718369 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c25e01d-b4b7-4198-a519-d1c6614e8893" containerName="istio-proxy" Nov 01 16:19:45.725132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.725110 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.729286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.729263 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb"] Nov 01 16:19:45.760459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.760427 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:19:45.760627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.760481 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:19:45.780059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.780030 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc5fr\" (UniqueName: \"kubernetes.io/projected/5e1a9310-3b27-4f6b-964f-c638a7793d3b-kube-api-access-rc5fr\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.780269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.780241 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.780467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.780452 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.882102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.882062 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.882305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.882165 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc5fr\" (UniqueName: \"kubernetes.io/projected/5e1a9310-3b27-4f6b-964f-c638a7793d3b-kube-api-access-rc5fr\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.882305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.882278 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.882984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.882955 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.884811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.884741 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.902034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.894151 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc5fr\" (UniqueName: \"kubernetes.io/projected/5e1a9310-3b27-4f6b-964f-c638a7793d3b-kube-api-access-rc5fr\") pod \"maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:45.945169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:45.945133 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:46.044362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.044279 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:46.047118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.047073 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:46.145970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.145810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:46.240770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.238904 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q"] Nov 01 16:19:46.240770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.239199 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" containerID="cri-o://2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9" gracePeriod=30 Nov 01 16:19:46.273348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.273292 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:46.529815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.528381 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb"] Nov 01 16:19:46.529815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.528671 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:46.810113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.810015 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:46.955764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.950240 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d"] Nov 01 16:19:46.955764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.955466 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:46.960482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.960451 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:46.963429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.963119 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d"] Nov 01 16:19:46.978245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.978116 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:46.978245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.978191 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rch8q\" (UniqueName: \"kubernetes.io/projected/470872a8-5ad4-41e5-bb96-737202abc8d7-kube-api-access-rch8q\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:46.978493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:46.978470 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:47.079806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.079703 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:47.080120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.079878 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:47.080120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.079968 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rch8q\" (UniqueName: \"kubernetes.io/projected/470872a8-5ad4-41e5-bb96-737202abc8d7-kube-api-access-rch8q\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:47.080580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.080551 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:47.083334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.083312 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:47.088718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.088677 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rch8q\" (UniqueName: \"kubernetes.io/projected/470872a8-5ad4-41e5-bb96-737202abc8d7-kube-api-access-rch8q\") pod \"maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:47.127640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.127609 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" event={"ID":"5e1a9310-3b27-4f6b-964f-c638a7793d3b","Type":"ContainerStarted","Data":"15d6696f14ab55c1a8ff7bc85a9abae5206ed4ac9031e6025cb0c465678deb52"} Nov 01 16:19:47.284607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.284586 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:47.578104 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:19:47.578058 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod470872a8_5ad4_41e5_bb96_737202abc8d7.slice/crio-fe60a09591f0bac6ba31b4ba28d658cfe044076e95d2c25848379afe799b907d WatchSource:0}: Error finding container fe60a09591f0bac6ba31b4ba28d658cfe044076e95d2c25848379afe799b907d: Status 404 returned error can't find the container with id fe60a09591f0bac6ba31b4ba28d658cfe044076e95d2c25848379afe799b907d Nov 01 16:19:47.579774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.579728 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d"] Nov 01 16:19:47.748559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:47.748526 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:48.137548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:48.137454 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" event={"ID":"470872a8-5ad4-41e5-bb96-737202abc8d7","Type":"ContainerStarted","Data":"bf76f3636a5a866e7a7dee22343b9e5135d99614e3390ffb5b4225243b68518a"} Nov 01 16:19:48.137548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:48.137515 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" event={"ID":"470872a8-5ad4-41e5-bb96-737202abc8d7","Type":"ContainerStarted","Data":"fe60a09591f0bac6ba31b4ba28d658cfe044076e95d2c25848379afe799b907d"} Nov 01 16:19:48.139317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:48.139292 4573 generic.go:356] "Generic (PLEG): container finished" podID="5e1a9310-3b27-4f6b-964f-c638a7793d3b" containerID="13c50a22ac4dcf6d5bc20f99e5bba79601484db69eb0d3593935d0db89a17279" exitCode=0 Nov 01 16:19:48.139452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:48.139369 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" event={"ID":"5e1a9310-3b27-4f6b-964f-c638a7793d3b","Type":"ContainerDied","Data":"13c50a22ac4dcf6d5bc20f99e5bba79601484db69eb0d3593935d0db89a17279"} Nov 01 16:19:48.154925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:48.154880 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" podStartSLOduration=2.154868814 podStartE2EDuration="2.154868814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:19:48.152049385 +0000 UTC m=+1925.855215343" watchObservedRunningTime="2026-11-01 16:19:48.154868814 +0000 UTC m=+1925.858034749" Nov 01 16:19:48.579549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:48.579506 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:49.149071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.149030 4573 generic.go:356] "Generic (PLEG): container finished" podID="470872a8-5ad4-41e5-bb96-737202abc8d7" containerID="bf76f3636a5a866e7a7dee22343b9e5135d99614e3390ffb5b4225243b68518a" exitCode=0 Nov 01 16:19:49.149517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.149115 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" event={"ID":"470872a8-5ad4-41e5-bb96-737202abc8d7","Type":"ContainerDied","Data":"bf76f3636a5a866e7a7dee22343b9e5135d99614e3390ffb5b4225243b68518a"} Nov 01 16:19:49.352787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.352764 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:49.437606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.437523 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1a9310-3b27-4f6b-964f-c638a7793d3b-kube-api-access-rc5fr\" (UniqueName: \"kubernetes.io/projected/5e1a9310-3b27-4f6b-964f-c638a7793d3b-kube-api-access-rc5fr\") pod \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " Nov 01 16:19:49.437834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.437685 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-xks-ca\") pod \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " Nov 01 16:19:49.437939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.437888 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-openshift-ca\") pod \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\" (UID: \"5e1a9310-3b27-4f6b-964f-c638a7793d3b\") " Nov 01 16:19:49.438288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.438261 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-openshift-ca" pod "5e1a9310-3b27-4f6b-964f-c638a7793d3b" (UID: "5e1a9310-3b27-4f6b-964f-c638a7793d3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:49.438524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.438507 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:49.439714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.439690 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1a9310-3b27-4f6b-964f-c638a7793d3b-kube-api-access-rc5fr" pod "5e1a9310-3b27-4f6b-964f-c638a7793d3b" (UID: "5e1a9310-3b27-4f6b-964f-c638a7793d3b"). InnerVolumeSpecName "kube-api-access-rc5fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:49.439835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.439786 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-xks-ca" pod "5e1a9310-3b27-4f6b-964f-c638a7793d3b" (UID: "5e1a9310-3b27-4f6b-964f-c638a7793d3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:49.447806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.447776 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:49.476822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.476787 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" podUID="b364e1c7-df3f-4daa-b582-d11057bd4585" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:49.476822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.476811 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" podUID="73f2b95e-884c-4b6c-8a93-4b47fccc47de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:49.543193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.541721 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc5fr\" (UniqueName: \"kubernetes.io/projected/5e1a9310-3b27-4f6b-964f-c638a7793d3b-kube-api-access-rc5fr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:49.543489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:49.543454 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e1a9310-3b27-4f6b-964f-c638a7793d3b-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.159924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.159894 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" Nov 01 16:19:50.160360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.159919 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb" event={"ID":"5e1a9310-3b27-4f6b-964f-c638a7793d3b","Type":"ContainerDied","Data":"15d6696f14ab55c1a8ff7bc85a9abae5206ed4ac9031e6025cb0c465678deb52"} Nov 01 16:19:50.160360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.159971 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15d6696f14ab55c1a8ff7bc85a9abae5206ed4ac9031e6025cb0c465678deb52" Nov 01 16:19:50.341951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.341924 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:50.354709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.354684 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470872a8-5ad4-41e5-bb96-737202abc8d7-kube-api-access-rch8q\" (UniqueName: \"kubernetes.io/projected/470872a8-5ad4-41e5-bb96-737202abc8d7-kube-api-access-rch8q\") pod \"470872a8-5ad4-41e5-bb96-737202abc8d7\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " Nov 01 16:19:50.355001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.354982 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-xks-ca\") pod \"470872a8-5ad4-41e5-bb96-737202abc8d7\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " Nov 01 16:19:50.355086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.355052 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-openshift-ca\") pod \"470872a8-5ad4-41e5-bb96-737202abc8d7\" (UID: \"470872a8-5ad4-41e5-bb96-737202abc8d7\") " Nov 01 16:19:50.355370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.355349 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-openshift-ca" pod "470872a8-5ad4-41e5-bb96-737202abc8d7" (UID: "470872a8-5ad4-41e5-bb96-737202abc8d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:50.355689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.355674 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.356727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.356703 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470872a8-5ad4-41e5-bb96-737202abc8d7-kube-api-access-rch8q" pod "470872a8-5ad4-41e5-bb96-737202abc8d7" (UID: "470872a8-5ad4-41e5-bb96-737202abc8d7"). InnerVolumeSpecName "kube-api-access-rch8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:50.356927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.356912 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-xks-ca" pod "470872a8-5ad4-41e5-bb96-737202abc8d7" (UID: "470872a8-5ad4-41e5-bb96-737202abc8d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:19:50.457292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.457259 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470872a8-5ad4-41e5-bb96-737202abc8d7-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.457417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.457297 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rch8q\" (UniqueName: \"kubernetes.io/projected/470872a8-5ad4-41e5-bb96-737202abc8d7-kube-api-access-rch8q\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.547422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.547384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:50.656662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.656052 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:19:50.761067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.760980 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-certs\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761076 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-envoy\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761117 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-socket\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761168 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-podinfo\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761227 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-token\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-token\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761282 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-certs" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.761397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761327 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-data\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761439 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3900c769-1183-45c0-aa46-0cb33ddc5778-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3900c769-1183-45c0-aa46-0cb33ddc5778-istiod-ca-cert\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761496 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-socket" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.761555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761551 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-credential-socket\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761614 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-data" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.761736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761664 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-kube-api-access-kd2kl\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-kube-api-access-kd2kl\") pod \"3900c769-1183-45c0-aa46-0cb33ddc5778\" (UID: \"3900c769-1183-45c0-aa46-0cb33ddc5778\") " Nov 01 16:19:50.761886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761861 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-credential-socket" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.762002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.761974 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3900c769-1183-45c0-aa46-0cb33ddc5778-istiod-ca-cert" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:50.762698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.762670 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.762873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.762703 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.762873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.762719 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.762873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.762733 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3900c769-1183-45c0-aa46-0cb33ddc5778-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.762873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.762746 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.763573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.763555 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-envoy" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:50.764179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.764159 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-podinfo" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:50.764268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.764217 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-token" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:50.764467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.764451 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-kube-api-access-kd2kl" pod "3900c769-1183-45c0-aa46-0cb33ddc5778" (UID: "3900c769-1183-45c0-aa46-0cb33ddc5778"). InnerVolumeSpecName "kube-api-access-kd2kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:50.864184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.864152 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.864184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.864180 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.864184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.864189 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.864184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.864198 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd2kl\" (UniqueName: \"kubernetes.io/projected/3900c769-1183-45c0-aa46-0cb33ddc5778-kube-api-access-kd2kl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:50.968244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:50.968209 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:51.061021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.060952 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:51.169642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.169612 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" Nov 01 16:19:51.169642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.169629 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d" event={"ID":"470872a8-5ad4-41e5-bb96-737202abc8d7","Type":"ContainerDied","Data":"fe60a09591f0bac6ba31b4ba28d658cfe044076e95d2c25848379afe799b907d"} Nov 01 16:19:51.170225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.169679 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe60a09591f0bac6ba31b4ba28d658cfe044076e95d2c25848379afe799b907d" Nov 01 16:19:51.171598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.171570 4573 generic.go:356] "Generic (PLEG): container finished" podID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerID="4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae" exitCode=0 Nov 01 16:19:51.171770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.171626 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" Nov 01 16:19:51.171770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.171654 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" event={"ID":"3900c769-1183-45c0-aa46-0cb33ddc5778","Type":"ContainerDied","Data":"4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae"} Nov 01 16:19:51.171770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.171709 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9" event={"ID":"3900c769-1183-45c0-aa46-0cb33ddc5778","Type":"ContainerDied","Data":"859c62e960cd94989836bc9db3b43b86f842a4d5501b68ea7b4c3f902beff82f"} Nov 01 16:19:51.171770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.171735 4573 scope.go:122] "RemoveContainer" containerID="4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae" Nov 01 16:19:51.183127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.183111 4573 scope.go:122] "RemoveContainer" containerID="61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33" Nov 01 16:19:51.193911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.193891 4573 scope.go:122] "RemoveContainer" containerID="4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae" Nov 01 16:19:51.194210 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:51.194191 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae\": container with ID starting with 4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae not found: ID does not exist" containerID="4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae" Nov 01 16:19:51.194278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.194217 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae"} err="failed to get container status \"4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae\": rpc error: code = NotFound desc = could not find container \"4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae\": container with ID starting with 4b11d32559e2546782b8a7cd2b683fc86b49e9f332f9979fd2bb73fccfd586ae not found: ID does not exist" Nov 01 16:19:51.194278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.194232 4573 scope.go:122] "RemoveContainer" containerID="61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33" Nov 01 16:19:51.194449 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:51.194435 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33\": container with ID starting with 61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33 not found: ID does not exist" containerID="61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33" Nov 01 16:19:51.194496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.194454 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33"} err="failed to get container status \"61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33\": rpc error: code = NotFound desc = could not find container \"61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33\": container with ID starting with 61354c50a71886050194260f1c41141d2d412a50864a109ae75cc54dbcc41c33 not found: ID does not exist" Nov 01 16:19:51.205012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.204989 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9"] Nov 01 16:19:51.212965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.212944 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2ce08a-openshift-default-9f569c5c5-6xrv9"] Nov 01 16:19:51.496856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.496817 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:51.525798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.525775 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:19:51.587882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.587450 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-podinfo\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.613093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.611608 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-podinfo" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:19:51.691360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691328 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-envoy\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.691561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691377 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-credential-socket\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.691561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691436 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996fafaf-f9d4-41bf-a519-6c3f4575b473-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996fafaf-f9d4-41bf-a519-6c3f4575b473-istiod-ca-cert\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.691804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691728 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-kube-api-access-nfwzm\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-kube-api-access-nfwzm\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.691804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691799 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-socket\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.691951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691874 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-certs\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.692020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691950 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-token\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-token\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.692020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691991 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-credential-socket" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.692020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.691999 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-data\") pod \"996fafaf-f9d4-41bf-a519-6c3f4575b473\" (UID: \"996fafaf-f9d4-41bf-a519-6c3f4575b473\") " Nov 01 16:19:51.692263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692240 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-socket" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.692403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692365 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996fafaf-f9d4-41bf-a519-6c3f4575b473-istiod-ca-cert" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:51.692476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692446 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-certs" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.692535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692499 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-data" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.692866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692846 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.692961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692868 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.692961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692878 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/996fafaf-f9d4-41bf-a519-6c3f4575b473-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.692961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692888 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.692961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692899 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.692961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.692912 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.694417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.694385 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-envoy" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:19:51.694526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.694483 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-kube-api-access-nfwzm" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "kube-api-access-nfwzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:51.695586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.695564 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-token" pod "996fafaf-f9d4-41bf-a519-6c3f4575b473" (UID: "996fafaf-f9d4-41bf-a519-6c3f4575b473"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:51.794292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.794259 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfwzm\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-kube-api-access-nfwzm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.794292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.794287 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:51.794292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:51.794297 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/996fafaf-f9d4-41bf-a519-6c3f4575b473-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:19:52.121412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.121372 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:52.184054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.184025 4573 generic.go:356] "Generic (PLEG): container finished" podID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerID="2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9" exitCode=0 Nov 01 16:19:52.184491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.184062 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" event={"ID":"996fafaf-f9d4-41bf-a519-6c3f4575b473","Type":"ContainerDied","Data":"2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9"} Nov 01 16:19:52.184491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.184089 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" Nov 01 16:19:52.184491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.184125 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q" event={"ID":"996fafaf-f9d4-41bf-a519-6c3f4575b473","Type":"ContainerDied","Data":"bd673ee52229559e8691e905d21c385074f9ea30dbe71a856bf3f6efe58ff76f"} Nov 01 16:19:52.184491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.184147 4573 scope.go:122] "RemoveContainer" containerID="2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9" Nov 01 16:19:52.196221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.196199 4573 scope.go:122] "RemoveContainer" containerID="092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2" Nov 01 16:19:52.209419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.208572 4573 scope.go:122] "RemoveContainer" containerID="2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9" Nov 01 16:19:52.209651 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:52.209623 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9\": container with ID starting with 2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9 not found: ID does not exist" containerID="2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9" Nov 01 16:19:52.209724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.209659 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9"} err="failed to get container status \"2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9\": rpc error: code = NotFound desc = could not find container \"2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9\": container with ID starting with 2e451f1fd7b6ae7b55701063f1314b211655f80ea7050d47093114a5f3890ff9 not found: ID does not exist" Nov 01 16:19:52.209724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.209677 4573 scope.go:122] "RemoveContainer" containerID="092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2" Nov 01 16:19:52.210051 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:52.209989 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2\": container with ID starting with 092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2 not found: ID does not exist" containerID="092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2" Nov 01 16:19:52.210051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.210022 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2"} err="failed to get container status \"092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2\": rpc error: code = NotFound desc = could not find container \"092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2\": container with ID starting with 092ea057af266df20c53877a888d1a700b80aabed722ddf9d1d67763d8e9b4a2 not found: ID does not exist" Nov 01 16:19:52.214218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.214197 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q"] Nov 01 16:19:52.221612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.221586 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:52.224799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.224775 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2001b0-openshift-default-ffb8fbb5c-jb68q"] Nov 01 16:19:52.561838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.559234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:52.578071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.577693 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" path="/var/lib/kubelet/pods/3900c769-1183-45c0-aa46-0cb33ddc5778/volumes" Nov 01 16:19:52.578511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:52.578486 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" path="/var/lib/kubelet/pods/996fafaf-f9d4-41bf-a519-6c3f4575b473/volumes" Nov 01 16:19:53.306900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:53.306864 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:54.093631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:54.093547 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:54.365047 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:19:54.365015 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:19:54.755616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:54.755513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:55.480786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:55.480743 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:55.879003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:55.878966 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:55.978101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:55.978028 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:56.469836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:56.469518 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:57.000103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:57.000069 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9"] Nov 01 16:19:57.015237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:57.015203 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq"] Nov 01 16:19:57.065539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:57.065502 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:57.159381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:57.159348 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:57.782030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:57.781973 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:58.549509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:58.549478 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:59.254415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.254384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:19:59.476744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.476706 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" podUID="b364e1c7-df3f-4daa-b582-d11057bd4585" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:59.476925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.476803 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" podUID="73f2b95e-884c-4b6c-8a93-4b47fccc47de" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:19:59.903274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.903245 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:19:59.940531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.940497 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:19:59.966064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.965995 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f2b95e-884c-4b6c-8a93-4b47fccc47de-kube-api-access-ndldr\" (UniqueName: \"kubernetes.io/projected/73f2b95e-884c-4b6c-8a93-4b47fccc47de-kube-api-access-ndldr\") pod \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\" (UID: \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\") " Nov 01 16:19:59.966170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.966066 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b364e1c7-df3f-4daa-b582-d11057bd4585-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364e1c7-df3f-4daa-b582-d11057bd4585-plugins-config-volume\") pod \"b364e1c7-df3f-4daa-b582-d11057bd4585\" (UID: \"b364e1c7-df3f-4daa-b582-d11057bd4585\") " Nov 01 16:19:59.966278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.966262 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b364e1c7-df3f-4daa-b582-d11057bd4585-kube-api-access-2f2gw\" (UniqueName: \"kubernetes.io/projected/b364e1c7-df3f-4daa-b582-d11057bd4585-kube-api-access-2f2gw\") pod \"b364e1c7-df3f-4daa-b582-d11057bd4585\" (UID: \"b364e1c7-df3f-4daa-b582-d11057bd4585\") " Nov 01 16:19:59.966430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.966414 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f2b95e-884c-4b6c-8a93-4b47fccc47de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f2b95e-884c-4b6c-8a93-4b47fccc47de-plugins-config-volume\") pod \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\" (UID: \"73f2b95e-884c-4b6c-8a93-4b47fccc47de\") " Nov 01 16:19:59.967217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.967191 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f2b95e-884c-4b6c-8a93-4b47fccc47de-plugins-config-volume" pod "73f2b95e-884c-4b6c-8a93-4b47fccc47de" (UID: "73f2b95e-884c-4b6c-8a93-4b47fccc47de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:59.967536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.967517 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b364e1c7-df3f-4daa-b582-d11057bd4585-plugins-config-volume" pod "b364e1c7-df3f-4daa-b582-d11057bd4585" (UID: "b364e1c7-df3f-4daa-b582-d11057bd4585"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:19:59.968280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.968254 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f2b95e-884c-4b6c-8a93-4b47fccc47de-kube-api-access-ndldr" pod "73f2b95e-884c-4b6c-8a93-4b47fccc47de" (UID: "73f2b95e-884c-4b6c-8a93-4b47fccc47de"). InnerVolumeSpecName "kube-api-access-ndldr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:59.969587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.969563 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b364e1c7-df3f-4daa-b582-d11057bd4585-kube-api-access-2f2gw" pod "b364e1c7-df3f-4daa-b582-d11057bd4585" (UID: "b364e1c7-df3f-4daa-b582-d11057bd4585"). InnerVolumeSpecName "kube-api-access-2f2gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:19:59.982549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:19:59.982510 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:00.077196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.068918 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f2gw\" (UniqueName: \"kubernetes.io/projected/b364e1c7-df3f-4daa-b582-d11057bd4585-kube-api-access-2f2gw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:00.077849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.077740 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73f2b95e-884c-4b6c-8a93-4b47fccc47de-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:00.078026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.078010 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndldr\" (UniqueName: \"kubernetes.io/projected/73f2b95e-884c-4b6c-8a93-4b47fccc47de-kube-api-access-ndldr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:00.078141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.078120 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b364e1c7-df3f-4daa-b582-d11057bd4585-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:00.380475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.380439 4573 generic.go:356] "Generic (PLEG): container finished" podID="b364e1c7-df3f-4daa-b582-d11057bd4585" containerID="33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e" exitCode=0 Nov 01 16:20:00.380672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.380508 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" Nov 01 16:20:00.380672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.380523 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" event={"ID":"b364e1c7-df3f-4daa-b582-d11057bd4585","Type":"ContainerDied","Data":"33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e"} Nov 01 16:20:00.380672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.380621 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2" event={"ID":"b364e1c7-df3f-4daa-b582-d11057bd4585","Type":"ContainerDied","Data":"71a2aa5715aa470f4046b3656deaebef955a3f2a492c23c480fbe8920a8f9d84"} Nov 01 16:20:00.380672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.380655 4573 scope.go:122] "RemoveContainer" containerID="33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e" Nov 01 16:20:00.382327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.382291 4573 generic.go:356] "Generic (PLEG): container finished" podID="73f2b95e-884c-4b6c-8a93-4b47fccc47de" containerID="e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2" exitCode=0 Nov 01 16:20:00.382327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.382321 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" event={"ID":"73f2b95e-884c-4b6c-8a93-4b47fccc47de","Type":"ContainerDied","Data":"e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2"} Nov 01 16:20:00.382532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.382347 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" Nov 01 16:20:00.382532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.382361 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g" event={"ID":"73f2b95e-884c-4b6c-8a93-4b47fccc47de","Type":"ContainerDied","Data":"5fbe403d6fd94a012814852735e06bfed98227e285d29a1e2574935572faf0ba"} Nov 01 16:20:00.392931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.392863 4573 scope.go:122] "RemoveContainer" containerID="33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e" Nov 01 16:20:00.393248 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:20:00.393221 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e\": container with ID starting with 33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e not found: ID does not exist" containerID="33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e" Nov 01 16:20:00.393316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.393261 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e"} err="failed to get container status \"33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e\": rpc error: code = NotFound desc = could not find container \"33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e\": container with ID starting with 33abd7a54985def4cc9f32aca7b75025940cae6455b74969e1750559299d8c9e not found: ID does not exist" Nov 01 16:20:00.393316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.393283 4573 scope.go:122] "RemoveContainer" containerID="e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2" Nov 01 16:20:00.403231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.403212 4573 scope.go:122] "RemoveContainer" containerID="e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2" Nov 01 16:20:00.403464 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:20:00.403445 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2\": container with ID starting with e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2 not found: ID does not exist" containerID="e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2" Nov 01 16:20:00.403513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.403471 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2"} err="failed to get container status \"e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2\": rpc error: code = NotFound desc = could not find container \"e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2\": container with ID starting with e52bb168b737ce0171625fd9eae9a1518b270aa1a7b00aef73cd201bcdf764e2 not found: ID does not exist" Nov 01 16:20:00.410458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.410437 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g"] Nov 01 16:20:00.414943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.414921 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2001b0-7f4d46cd5f-zlt2g"] Nov 01 16:20:00.427208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.427180 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2"] Nov 01 16:20:00.431349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.431324 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-2001b0-86fcc565f-jqsc2"] Nov 01 16:20:00.460256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.460231 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f2b95e-884c-4b6c-8a93-4b47fccc47de" path="/var/lib/kubelet/pods/73f2b95e-884c-4b6c-8a93-4b47fccc47de/volumes" Nov 01 16:20:00.460738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.460723 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b364e1c7-df3f-4daa-b582-d11057bd4585" path="/var/lib/kubelet/pods/b364e1c7-df3f-4daa-b582-d11057bd4585/volumes" Nov 01 16:20:00.667070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:00.666984 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:01.345359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:01.345328 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:02.000918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:02.000856 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" podUID="da65f65f-be21-4a00-bb5c-f0c3f183aee1" containerName="payload-processing" containerID="cri-o://d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e" gracePeriod=25 Nov 01 16:20:02.016541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:02.016505 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" podUID="c73f29d0-5265-499b-af4c-a47dec1c99dd" containerName="payload-pre-processing" containerID="cri-o://2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a" gracePeriod=25 Nov 01 16:20:02.299807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:02.299679 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:02.967929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:02.967892 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:03.623227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:03.623196 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:04.183892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:04.183862 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:04.333017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:04.332983 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:04.720979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:04.720922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:04.721185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:04.721014 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb"] Nov 01 16:20:04.726501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:04.726478 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99afca-8j4tb"] Nov 01 16:20:04.819077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:04.819044 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:05.160990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:05.160947 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:05.548242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:05.548151 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:06.071968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.071227 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps"] Nov 01 16:20:06.071968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.071502 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" podUID="18a0ee2f-1449-4e29-9d6b-084fb968a2dd" containerName="maas-api" containerID="cri-o://f33a37bc5e082892fc94a04a3de30d2768b1cf63df4edc39bb521a705570be33" gracePeriod=30 Nov 01 16:20:06.098410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.098366 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:06.247888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.247854 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:06.553307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.553276 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:06.572213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.572183 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1a9310-3b27-4f6b-964f-c638a7793d3b" path="/var/lib/kubelet/pods/5e1a9310-3b27-4f6b-964f-c638a7793d3b/volumes" Nov 01 16:20:06.575557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.575517 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d"] Nov 01 16:20:06.575743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.575730 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99e837-9rr6d"] Nov 01 16:20:06.580776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.580321 4573 generic.go:356] "Generic (PLEG): container finished" podID="18a0ee2f-1449-4e29-9d6b-084fb968a2dd" containerID="f33a37bc5e082892fc94a04a3de30d2768b1cf63df4edc39bb521a705570be33" exitCode=0 Nov 01 16:20:06.580776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.580464 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" event={"ID":"18a0ee2f-1449-4e29-9d6b-084fb968a2dd","Type":"ContainerDied","Data":"f33a37bc5e082892fc94a04a3de30d2768b1cf63df4edc39bb521a705570be33"} Nov 01 16:20:06.624833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.624576 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:06.725249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.725215 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:20:06.856366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.856269 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-kube-api-access-bm76p\" (UniqueName: \"kubernetes.io/projected/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-kube-api-access-bm76p\") pod \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " Nov 01 16:20:06.856366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.856353 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls\") pod \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " Nov 01 16:20:06.856601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.856432 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-openshift-service-ca\") pod \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " Nov 01 16:20:06.856601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.856547 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-metrics-certs\") pod \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\" (UID: \"18a0ee2f-1449-4e29-9d6b-084fb968a2dd\") " Nov 01 16:20:06.862169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.862138 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-metrics-certs" pod "18a0ee2f-1449-4e29-9d6b-084fb968a2dd" (UID: "18a0ee2f-1449-4e29-9d6b-084fb968a2dd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:06.872785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.865883 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls" pod "18a0ee2f-1449-4e29-9d6b-084fb968a2dd" (UID: "18a0ee2f-1449-4e29-9d6b-084fb968a2dd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:06.872785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.868915 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-openshift-service-ca" pod "18a0ee2f-1449-4e29-9d6b-084fb968a2dd" (UID: "18a0ee2f-1449-4e29-9d6b-084fb968a2dd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:06.872785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.872160 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-kube-api-access-bm76p" pod "18a0ee2f-1449-4e29-9d6b-084fb968a2dd" (UID: "18a0ee2f-1449-4e29-9d6b-084fb968a2dd"). InnerVolumeSpecName "kube-api-access-bm76p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:06.959395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.959357 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:06.959395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.959399 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:06.959598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.959416 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm76p\" (UniqueName: \"kubernetes.io/projected/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-kube-api-access-bm76p\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:06.959598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.959433 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18a0ee2f-1449-4e29-9d6b-084fb968a2dd-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:06.982245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:06.982212 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:07.375512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:07.375477 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:07.591041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:07.591013 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" Nov 01 16:20:07.591492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:07.591046 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps" event={"ID":"18a0ee2f-1449-4e29-9d6b-084fb968a2dd","Type":"ContainerDied","Data":"7a65d2a6585fa47b5444414e9fc7e7a6c39130307863781f8748b3e035dfee8a"} Nov 01 16:20:07.591492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:07.591104 4573 scope.go:122] "RemoveContainer" containerID="f33a37bc5e082892fc94a04a3de30d2768b1cf63df4edc39bb521a705570be33" Nov 01 16:20:07.615367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:07.615325 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps"] Nov 01 16:20:07.620087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:07.620061 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-99e837-b79444d9d-njlps"] Nov 01 16:20:08.133083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:08.133052 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:08.235379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:08.235351 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:08.337355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:08.337314 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:08.460317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:08.460230 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a0ee2f-1449-4e29-9d6b-084fb968a2dd" path="/var/lib/kubelet/pods/18a0ee2f-1449-4e29-9d6b-084fb968a2dd/volumes" Nov 01 16:20:08.460830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:08.460813 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470872a8-5ad4-41e5-bb96-737202abc8d7" path="/var/lib/kubelet/pods/470872a8-5ad4-41e5-bb96-737202abc8d7/volumes" Nov 01 16:20:08.545680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:08.545650 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" podUID="c73f29d0-5265-499b-af4c-a47dec1c99dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:20:08.946958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:08.946906 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:09.678422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:09.678373 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:09.943327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:09.943234 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" podUID="da65f65f-be21-4a00-bb5c-f0c3f183aee1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:20:10.048875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.048835 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf"] Nov 01 16:20:10.049741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049716 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5e1a9310-3b27-4f6b-964f-c638a7793d3b" containerName="revoke-keys" Nov 01 16:20:10.049741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049741 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1a9310-3b27-4f6b-964f-c638a7793d3b" containerName="revoke-keys" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049775 4573 cpu_manager.go:414] "containerMap: removing container" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049782 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049795 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b364e1c7-df3f-4daa-b582-d11057bd4585" containerName="payload-pre-processing" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049802 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b364e1c7-df3f-4daa-b582-d11057bd4585" containerName="payload-pre-processing" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049824 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049831 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049853 4573 cpu_manager.go:414] "containerMap: removing container" podUID="18a0ee2f-1449-4e29-9d6b-084fb968a2dd" containerName="maas-api" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049860 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a0ee2f-1449-4e29-9d6b-084fb968a2dd" containerName="maas-api" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049874 4573 cpu_manager.go:414] "containerMap: removing container" podUID="470872a8-5ad4-41e5-bb96-737202abc8d7" containerName="revoke-keys" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049881 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470872a8-5ad4-41e5-bb96-737202abc8d7" containerName="revoke-keys" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049911 4573 cpu_manager.go:414] "containerMap: removing container" podUID="73f2b95e-884c-4b6c-8a93-4b47fccc47de" containerName="payload-processing" Nov 01 16:20:10.049950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.049919 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f2b95e-884c-4b6c-8a93-4b47fccc47de" containerName="payload-processing" Nov 01 16:20:10.050553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.050136 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1a9310-3b27-4f6b-964f-c638a7793d3b" containerName="revoke-keys" Nov 01 16:20:10.050553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.050154 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" Nov 01 16:20:10.050553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.050166 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b364e1c7-df3f-4daa-b582-d11057bd4585" containerName="payload-pre-processing" Nov 01 16:20:10.050553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.050177 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" Nov 01 16:20:10.050553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.050186 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" Nov 01 16:20:10.050553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.050197 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a0ee2f-1449-4e29-9d6b-084fb968a2dd" containerName="maas-api" Nov 01 16:20:10.050553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.050208 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="470872a8-5ad4-41e5-bb96-737202abc8d7" containerName="revoke-keys" Nov 01 16:20:10.050553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.050219 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f2b95e-884c-4b6c-8a93-4b47fccc47de" containerName="payload-processing" Nov 01 16:20:10.055948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.055923 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.059746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.059715 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:20:10.059902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.059879 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:20:10.060187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.060168 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf"] Nov 01 16:20:10.115079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.115034 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9zrj\" (UniqueName: \"kubernetes.io/projected/a5cd9e54-98f6-42f0-8ad3-af7939680840-kube-api-access-h9zrj\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.115284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.115206 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.115403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.115383 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.216874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.216779 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zrj\" (UniqueName: \"kubernetes.io/projected/a5cd9e54-98f6-42f0-8ad3-af7939680840-kube-api-access-h9zrj\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.217049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.216933 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.217134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.217113 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.218038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.217904 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.252487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.251674 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.265936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.265899 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zrj\" (UniqueName: \"kubernetes.io/projected/a5cd9e54-98f6-42f0-8ad3-af7939680840-kube-api-access-h9zrj\") pod \"maas-api-revoke-sub-09fdf186-7lfpf\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.374665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.374638 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:10.464907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.464867 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:10.659248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.659221 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf"] Nov 01 16:20:10.659690 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:20:10.659665 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cd9e54_98f6_42f0_8ad3_af7939680840.slice/crio-e2390374290bbe36fdd61cc63ac1f668e43cb2ef3f57d00f88b2f5d136e1b803 WatchSource:0}: Error finding container e2390374290bbe36fdd61cc63ac1f668e43cb2ef3f57d00f88b2f5d136e1b803: Status 404 returned error can't find the container with id e2390374290bbe36fdd61cc63ac1f668e43cb2ef3f57d00f88b2f5d136e1b803 Nov 01 16:20:10.687577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:10.687544 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" event={"ID":"a5cd9e54-98f6-42f0-8ad3-af7939680840","Type":"ContainerStarted","Data":"e2390374290bbe36fdd61cc63ac1f668e43cb2ef3f57d00f88b2f5d136e1b803"} Nov 01 16:20:11.079433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:11.079362 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:11.283185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:11.283152 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:11.392746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:11.392699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:11.590560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:11.590521 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62"] Nov 01 16:20:11.591161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:11.590809 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" containerID="cri-o://3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22" gracePeriod=30 Nov 01 16:20:11.602258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:11.602217 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:11.894822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:11.894589 4573 generic.go:356] "Generic (PLEG): container finished" podID="a5cd9e54-98f6-42f0-8ad3-af7939680840" containerID="6eecd7372de6777e4dfaf1e342665f84a95e07160538bb88405f3c0903923299" exitCode=0 Nov 01 16:20:11.895012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:11.894913 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" event={"ID":"a5cd9e54-98f6-42f0-8ad3-af7939680840","Type":"ContainerDied","Data":"6eecd7372de6777e4dfaf1e342665f84a95e07160538bb88405f3c0903923299"} Nov 01 16:20:12.053230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:12.053142 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:12.676277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:12.676240 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:13.043425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.043377 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:13.248956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.248925 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:13.259304 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.259275 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-openshift-ca\") pod \"a5cd9e54-98f6-42f0-8ad3-af7939680840\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " Nov 01 16:20:13.259446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.259360 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-xks-ca\") pod \"a5cd9e54-98f6-42f0-8ad3-af7939680840\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " Nov 01 16:20:13.259551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.259537 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5cd9e54-98f6-42f0-8ad3-af7939680840-kube-api-access-h9zrj\" (UniqueName: \"kubernetes.io/projected/a5cd9e54-98f6-42f0-8ad3-af7939680840-kube-api-access-h9zrj\") pod \"a5cd9e54-98f6-42f0-8ad3-af7939680840\" (UID: \"a5cd9e54-98f6-42f0-8ad3-af7939680840\") " Nov 01 16:20:13.260742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.260717 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-openshift-ca" pod "a5cd9e54-98f6-42f0-8ad3-af7939680840" (UID: "a5cd9e54-98f6-42f0-8ad3-af7939680840"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:13.262249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.262226 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5cd9e54-98f6-42f0-8ad3-af7939680840-kube-api-access-h9zrj" pod "a5cd9e54-98f6-42f0-8ad3-af7939680840" (UID: "a5cd9e54-98f6-42f0-8ad3-af7939680840"). InnerVolumeSpecName "kube-api-access-h9zrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:13.262616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.262594 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-xks-ca" pod "a5cd9e54-98f6-42f0-8ad3-af7939680840" (UID: "a5cd9e54-98f6-42f0-8ad3-af7939680840"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:13.294553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.294510 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:13.366904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.364295 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9zrj\" (UniqueName: \"kubernetes.io/projected/a5cd9e54-98f6-42f0-8ad3-af7939680840-kube-api-access-h9zrj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:13.366904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.364330 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:13.366904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.364345 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5cd9e54-98f6-42f0-8ad3-af7939680840-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:13.395326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.395292 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:13.599101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.599023 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9"] Nov 01 16:20:13.599336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.599279 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" containerID="cri-o://23fd6971e35bb8db474a03f323737e27ddb38084dcbeebeee4f66ebeac7f83cb" gracePeriod=30 Nov 01 16:20:13.623871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.623840 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:13.899834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.891509 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:13.939318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.939286 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" Nov 01 16:20:13.939318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.939310 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf" event={"ID":"a5cd9e54-98f6-42f0-8ad3-af7939680840","Type":"ContainerDied","Data":"e2390374290bbe36fdd61cc63ac1f668e43cb2ef3f57d00f88b2f5d136e1b803"} Nov 01 16:20:13.939587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:13.939360 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2390374290bbe36fdd61cc63ac1f668e43cb2ef3f57d00f88b2f5d136e1b803" Nov 01 16:20:14.319509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:14.319425 4573 patch_prober.go:29] interesting pod/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:20:14.319509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:14.319477 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:20:14.689453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:14.689419 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:15.482523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:15.482490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:16.351257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.351220 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:16.885482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.885458 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:20:16.933308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933276 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-data\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.933499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933319 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-envoy\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.933578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933500 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-credential-socket\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.933578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933523 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-socket\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.933578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933553 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84e67049-8316-4cef-8d25-3e09e68797fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e67049-8316-4cef-8d25-3e09e68797fe-istio-podinfo\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.933840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933612 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e67049-8316-4cef-8d25-3e09e68797fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e67049-8316-4cef-8d25-3e09e68797fe-istiod-ca-cert\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.933840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933616 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-data" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:16.933840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933647 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-certs\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.933840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933818 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-credential-socket" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:16.933840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933831 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-kube-api-access-st6nq\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-kube-api-access-st6nq\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.934123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.933870 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-istio-token\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-istio-token\") pod \"84e67049-8316-4cef-8d25-3e09e68797fe\" (UID: \"84e67049-8316-4cef-8d25-3e09e68797fe\") " Nov 01 16:20:16.934123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.934030 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-socket" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:16.934238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.934186 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e67049-8316-4cef-8d25-3e09e68797fe-istiod-ca-cert" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:16.934238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.934218 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-certs" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:16.934956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.934806 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:16.934956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.934835 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:16.934956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.934853 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e67049-8316-4cef-8d25-3e09e68797fe-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:16.934956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.934869 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:16.934956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.934884 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:16.936446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.936413 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84e67049-8316-4cef-8d25-3e09e68797fe-istio-podinfo" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:20:16.936687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.936663 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-istio-token" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:16.936872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.936847 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-envoy" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:16.937860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.937827 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-kube-api-access-st6nq" pod "84e67049-8316-4cef-8d25-3e09e68797fe" (UID: "84e67049-8316-4cef-8d25-3e09e68797fe"). InnerVolumeSpecName "kube-api-access-st6nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:16.967992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.967961 4573 generic.go:356] "Generic (PLEG): container finished" podID="84e67049-8316-4cef-8d25-3e09e68797fe" containerID="3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22" exitCode=0 Nov 01 16:20:16.968136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.968061 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" event={"ID":"84e67049-8316-4cef-8d25-3e09e68797fe","Type":"ContainerDied","Data":"3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22"} Nov 01 16:20:16.968136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.968081 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" Nov 01 16:20:16.968136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.968120 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62" event={"ID":"84e67049-8316-4cef-8d25-3e09e68797fe","Type":"ContainerDied","Data":"6131bcd7ab8a960d5cb8dc6b21ad34b348c7181091d7162f141df6f77598b93f"} Nov 01 16:20:16.968305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.968151 4573 scope.go:122] "RemoveContainer" containerID="3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22" Nov 01 16:20:16.981138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.981108 4573 scope.go:122] "RemoveContainer" containerID="3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22" Nov 01 16:20:16.981418 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:20:16.981392 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22\": container with ID starting with 3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22 not found: ID does not exist" containerID="3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22" Nov 01 16:20:16.981466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.981426 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22"} err="failed to get container status \"3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22\": rpc error: code = NotFound desc = could not find container \"3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22\": container with ID starting with 3e1bf49a65d7c20966115525f376d140884a96231bceba3daf5e5d8b5c8a5e22 not found: ID does not exist" Nov 01 16:20:16.997248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:16.997219 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62"] Nov 01 16:20:17.005072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:17.004430 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-99afca-openshift-default-84d498d4ff-8ng62"] Nov 01 16:20:17.036393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:17.036351 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e67049-8316-4cef-8d25-3e09e68797fe-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.036393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:17.036393 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st6nq\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-kube-api-access-st6nq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.036604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:17.036410 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e67049-8316-4cef-8d25-3e09e68797fe-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.036604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:17.036424 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e67049-8316-4cef-8d25-3e09e68797fe-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:17.187301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:17.187218 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:18.218030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:18.217998 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:18.460124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:18.460092 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" path="/var/lib/kubelet/pods/84e67049-8316-4cef-8d25-3e09e68797fe/volumes" Nov 01 16:20:18.545324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:18.545240 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" podUID="c73f29d0-5265-499b-af4c-a47dec1c99dd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:20:18.914402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:18.914359 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5"] Nov 01 16:20:18.914593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:18.914549 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" containerID="cri-o://e4e5b265dea7a8c03af10ef341b69ce376670a0ee8108c75e4b3631c3d682e1b" gracePeriod=30 Nov 01 16:20:18.986281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:18.986244 4573 generic.go:356] "Generic (PLEG): container finished" podID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerID="23fd6971e35bb8db474a03f323737e27ddb38084dcbeebeee4f66ebeac7f83cb" exitCode=0 Nov 01 16:20:18.986281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:18.986280 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" event={"ID":"0dfdff89-3026-4c1e-b2f1-3d77ace7963b","Type":"ContainerDied","Data":"23fd6971e35bb8db474a03f323737e27ddb38084dcbeebeee4f66ebeac7f83cb"} Nov 01 16:20:19.221411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.221371 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:19.331741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.331708 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:20:19.374526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.374494 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-envoy\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.374703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.374560 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-podinfo\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.374703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.374632 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istiod-ca-cert\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.374703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.374665 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-kube-api-access-9gz5k\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-kube-api-access-9gz5k\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.375000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.374976 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-socket\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.375094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.375049 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-certs\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.375168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.375104 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-data\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.375168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.375121 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istiod-ca-cert" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:19.375168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.375150 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-credential-socket\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.375408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.375181 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-token\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-token\") pod \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\" (UID: \"0dfdff89-3026-4c1e-b2f1-3d77ace7963b\") " Nov 01 16:20:19.375847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.375810 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-socket" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.375962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.375946 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.376029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.375970 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.376094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.376061 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-certs" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.376160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.376108 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-credential-socket" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.376219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.376177 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-data" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.377348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.377322 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-envoy" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:19.378174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.378148 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-token" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:19.378334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.378313 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-kube-api-access-9gz5k" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "kube-api-access-9gz5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:19.378444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.378425 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-podinfo" pod "0dfdff89-3026-4c1e-b2f1-3d77ace7963b" (UID: "0dfdff89-3026-4c1e-b2f1-3d77ace7963b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:20:19.477254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.477177 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.477254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.477205 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.477254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.477216 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.477254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.477225 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.477254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.477234 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.477254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.477243 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.477254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.477257 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gz5k\" (UniqueName: \"kubernetes.io/projected/0dfdff89-3026-4c1e-b2f1-3d77ace7963b-kube-api-access-9gz5k\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:19.943568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.943526 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" podUID="da65f65f-be21-4a00-bb5c-f0c3f183aee1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:20:19.995236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.995207 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" Nov 01 16:20:19.995406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.995234 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9" event={"ID":"0dfdff89-3026-4c1e-b2f1-3d77ace7963b","Type":"ContainerDied","Data":"93d86b265d7d7f19593c2f486cfaf0e7ba6f39c7c1675c869cdbf8ef2db53ec7"} Nov 01 16:20:19.995406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:19.995300 4573 scope.go:122] "RemoveContainer" containerID="23fd6971e35bb8db474a03f323737e27ddb38084dcbeebeee4f66ebeac7f83cb" Nov 01 16:20:20.036315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:20.036274 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9"] Nov 01 16:20:20.041261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:20.041237 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-99e837-openshift-default-7bc77d59c5-hlfr9"] Nov 01 16:20:20.094439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:20.094410 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:20.461468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:20.461432 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" path="/var/lib/kubelet/pods/0dfdff89-3026-4c1e-b2f1-3d77ace7963b/volumes" Nov 01 16:20:21.095089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:21.095039 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:22.074834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.074797 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:22.077459 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:20:22.077433 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda65f65f_be21_4a00_bb5c_f0c3f183aee1.slice/crio-d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73f29d0_5265_499b_af4c_a47dec1c99dd.slice/crio-conmon-2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc73f29d0_5265_499b_af4c_a47dec1c99dd.slice/crio-2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:20:22.464067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.464036 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:20:22.472555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.472529 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:20:22.554621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.554588 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c73f29d0-5265-499b-af4c-a47dec1c99dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73f29d0-5265-499b-af4c-a47dec1c99dd-plugins-config-volume\") pod \"c73f29d0-5265-499b-af4c-a47dec1c99dd\" (UID: \"c73f29d0-5265-499b-af4c-a47dec1c99dd\") " Nov 01 16:20:22.554827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.554804 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da65f65f-be21-4a00-bb5c-f0c3f183aee1-kube-api-access-bhfvt\" (UniqueName: \"kubernetes.io/projected/da65f65f-be21-4a00-bb5c-f0c3f183aee1-kube-api-access-bhfvt\") pod \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\" (UID: \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\") " Nov 01 16:20:22.554939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.554918 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c73f29d0-5265-499b-af4c-a47dec1c99dd-kube-api-access-67n2t\" (UniqueName: \"kubernetes.io/projected/c73f29d0-5265-499b-af4c-a47dec1c99dd-kube-api-access-67n2t\") pod \"c73f29d0-5265-499b-af4c-a47dec1c99dd\" (UID: \"c73f29d0-5265-499b-af4c-a47dec1c99dd\") " Nov 01 16:20:22.555005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.554987 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da65f65f-be21-4a00-bb5c-f0c3f183aee1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da65f65f-be21-4a00-bb5c-f0c3f183aee1-plugins-config-volume\") pod \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\" (UID: \"da65f65f-be21-4a00-bb5c-f0c3f183aee1\") " Nov 01 16:20:22.555063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.554994 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c73f29d0-5265-499b-af4c-a47dec1c99dd-plugins-config-volume" pod "c73f29d0-5265-499b-af4c-a47dec1c99dd" (UID: "c73f29d0-5265-499b-af4c-a47dec1c99dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:22.555380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.555358 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da65f65f-be21-4a00-bb5c-f0c3f183aee1-plugins-config-volume" pod "da65f65f-be21-4a00-bb5c-f0c3f183aee1" (UID: "da65f65f-be21-4a00-bb5c-f0c3f183aee1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:22.555860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.555837 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da65f65f-be21-4a00-bb5c-f0c3f183aee1-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:22.555991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.555865 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c73f29d0-5265-499b-af4c-a47dec1c99dd-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:22.557209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.557187 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c73f29d0-5265-499b-af4c-a47dec1c99dd-kube-api-access-67n2t" pod "c73f29d0-5265-499b-af4c-a47dec1c99dd" (UID: "c73f29d0-5265-499b-af4c-a47dec1c99dd"). InnerVolumeSpecName "kube-api-access-67n2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:22.557314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.557221 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da65f65f-be21-4a00-bb5c-f0c3f183aee1-kube-api-access-bhfvt" pod "da65f65f-be21-4a00-bb5c-f0c3f183aee1" (UID: "da65f65f-be21-4a00-bb5c-f0c3f183aee1"). InnerVolumeSpecName "kube-api-access-bhfvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:22.657421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.657380 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhfvt\" (UniqueName: \"kubernetes.io/projected/da65f65f-be21-4a00-bb5c-f0c3f183aee1-kube-api-access-bhfvt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:22.657421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.657420 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67n2t\" (UniqueName: \"kubernetes.io/projected/c73f29d0-5265-499b-af4c-a47dec1c99dd-kube-api-access-67n2t\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:22.867816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:22.867706 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:23.025255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.025226 4573 generic.go:356] "Generic (PLEG): container finished" podID="da65f65f-be21-4a00-bb5c-f0c3f183aee1" containerID="d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e" exitCode=0 Nov 01 16:20:23.025416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.025277 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" Nov 01 16:20:23.025416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.025313 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" event={"ID":"da65f65f-be21-4a00-bb5c-f0c3f183aee1","Type":"ContainerDied","Data":"d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e"} Nov 01 16:20:23.025416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.025373 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9" event={"ID":"da65f65f-be21-4a00-bb5c-f0c3f183aee1","Type":"ContainerDied","Data":"897f588950ad57f94221e7decfee521497fe26afbd42db8b6261a44e4e641b4a"} Nov 01 16:20:23.025416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.025402 4573 scope.go:122] "RemoveContainer" containerID="d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e" Nov 01 16:20:23.026875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.026851 4573 generic.go:356] "Generic (PLEG): container finished" podID="c73f29d0-5265-499b-af4c-a47dec1c99dd" containerID="2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a" exitCode=0 Nov 01 16:20:23.026997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.026882 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" event={"ID":"c73f29d0-5265-499b-af4c-a47dec1c99dd","Type":"ContainerDied","Data":"2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a"} Nov 01 16:20:23.026997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.026940 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" event={"ID":"c73f29d0-5265-499b-af4c-a47dec1c99dd","Type":"ContainerDied","Data":"45c5d7a04528696a6f0eb08987056ddc6d8071f301a2b0a7e1c0fa31f9cd383c"} Nov 01 16:20:23.026997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.026893 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq" Nov 01 16:20:23.037582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.037564 4573 scope.go:122] "RemoveContainer" containerID="d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e" Nov 01 16:20:23.037883 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:20:23.037864 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e\": container with ID starting with d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e not found: ID does not exist" containerID="d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e" Nov 01 16:20:23.037955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.037890 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e"} err="failed to get container status \"d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e\": rpc error: code = NotFound desc = could not find container \"d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e\": container with ID starting with d5abd4c56dfd54176daa920e05699d8526b2e37aeb412702783f5205d275277e not found: ID does not exist" Nov 01 16:20:23.037955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.037906 4573 scope.go:122] "RemoveContainer" containerID="2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a" Nov 01 16:20:23.048806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.048784 4573 scope.go:122] "RemoveContainer" containerID="2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a" Nov 01 16:20:23.049118 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:20:23.049096 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a\": container with ID starting with 2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a not found: ID does not exist" containerID="2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a" Nov 01 16:20:23.049184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.049123 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a"} err="failed to get container status \"2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a\": rpc error: code = NotFound desc = could not find container \"2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a\": container with ID starting with 2150fc20b44713553ba761515b9525222b24c8918e1af3a90162e8a4e189bb1a not found: ID does not exist" Nov 01 16:20:23.056914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.056893 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9"] Nov 01 16:20:23.062780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.062761 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99e837-75d8d9f76f-5r7g9"] Nov 01 16:20:23.080301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.080280 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq"] Nov 01 16:20:23.085932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.085913 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-99e837-84bbb74c87-vt2jq"] Nov 01 16:20:23.749362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:23.749327 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:24.037959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.037928 4573 generic.go:356] "Generic (PLEG): container finished" podID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerID="e4e5b265dea7a8c03af10ef341b69ce376670a0ee8108c75e4b3631c3d682e1b" exitCode=0 Nov 01 16:20:24.038117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.038001 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" event={"ID":"37db937e-ba95-48cd-ac9d-d93d4c28d0cf","Type":"ContainerDied","Data":"e4e5b265dea7a8c03af10ef341b69ce376670a0ee8108c75e4b3631c3d682e1b"} Nov 01 16:20:24.208454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.208428 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 16:20:24.290510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290421 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-certs\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.290675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290527 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-envoy\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.290675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290596 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-data\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.290675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290628 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-token\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-token\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.290675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290669 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istiod-ca-cert\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.290910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290698 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-kube-api-access-vpqd9\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-kube-api-access-vpqd9\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.290910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290762 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-certs" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:24.291014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290924 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-credential-socket\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.291014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.290967 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-socket\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.291130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291112 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-podinfo\") pod \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\" (UID: \"37db937e-ba95-48cd-ac9d-d93d4c28d0cf\") " Nov 01 16:20:24.291211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291193 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-credential-socket" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:24.291576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291296 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-socket" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:24.291698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291580 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istiod-ca-cert" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:24.291903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291823 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.291903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291848 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.291903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291863 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.291903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291877 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.292137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.291931 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-data" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:24.293599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.293250 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-envoy" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:20:24.293599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.293443 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-token" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:24.293722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.293608 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-podinfo" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:20:24.294011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.293988 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-kube-api-access-vpqd9" pod "37db937e-ba95-48cd-ac9d-d93d4c28d0cf" (UID: "37db937e-ba95-48cd-ac9d-d93d4c28d0cf"). InnerVolumeSpecName "kube-api-access-vpqd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:24.371786 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:20:24.371717 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:20:24.393689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.393659 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.393689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.393687 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.393689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.393697 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.393926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.393706 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpqd9\" (UniqueName: \"kubernetes.io/projected/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-kube-api-access-vpqd9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.393926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.393716 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37db937e-ba95-48cd-ac9d-d93d4c28d0cf-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:24.455533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.455499 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:24.460263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.460231 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c73f29d0-5265-499b-af4c-a47dec1c99dd" path="/var/lib/kubelet/pods/c73f29d0-5265-499b-af4c-a47dec1c99dd/volumes" Nov 01 16:20:24.460779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:24.460764 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da65f65f-be21-4a00-bb5c-f0c3f183aee1" path="/var/lib/kubelet/pods/da65f65f-be21-4a00-bb5c-f0c3f183aee1/volumes" Nov 01 16:20:25.050646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:25.050613 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" Nov 01 16:20:25.050646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:25.050643 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5" event={"ID":"37db937e-ba95-48cd-ac9d-d93d4c28d0cf","Type":"ContainerDied","Data":"80ed4feb896a3e0000b32938082761fd4174419031ff13d0890615d8dd2b1b16"} Nov 01 16:20:25.050941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:25.050701 4573 scope.go:122] "RemoveContainer" containerID="e4e5b265dea7a8c03af10ef341b69ce376670a0ee8108c75e4b3631c3d682e1b" Nov 01 16:20:25.072031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:25.072007 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5"] Nov 01 16:20:25.076895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:25.076874 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-kkcz5"] Nov 01 16:20:25.175483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:25.175445 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:25.872081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:25.872048 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:26.459482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:26.459440 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" path="/var/lib/kubelet/pods/37db937e-ba95-48cd-ac9d-d93d4c28d0cf/volumes" Nov 01 16:20:26.564979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:26.564951 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:27.242925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:27.242875 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:27.680613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:27.680573 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:28.219904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.219874 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:28.338854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.338816 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:28.776430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.776393 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5"] Nov 01 16:20:28.777183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777163 4573 cpu_manager.go:414] "containerMap: removing container" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" Nov 01 16:20:28.777243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777184 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" Nov 01 16:20:28.777243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777208 4573 cpu_manager.go:414] "containerMap: removing container" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" Nov 01 16:20:28.777243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777213 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996fafaf-f9d4-41bf-a519-6c3f4575b473" containerName="istio-proxy" Nov 01 16:20:28.777243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777223 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c73f29d0-5265-499b-af4c-a47dec1c99dd" containerName="payload-pre-processing" Nov 01 16:20:28.777243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777228 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c73f29d0-5265-499b-af4c-a47dec1c99dd" containerName="payload-pre-processing" Nov 01 16:20:28.777243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777238 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a5cd9e54-98f6-42f0-8ad3-af7939680840" containerName="revoke-keys" Nov 01 16:20:28.777243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777242 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5cd9e54-98f6-42f0-8ad3-af7939680840" containerName="revoke-keys" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777250 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777255 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777267 4573 cpu_manager.go:414] "containerMap: removing container" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777271 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777286 4573 cpu_manager.go:414] "containerMap: removing container" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777299 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777315 4573 cpu_manager.go:414] "containerMap: removing container" podUID="da65f65f-be21-4a00-bb5c-f0c3f183aee1" containerName="payload-processing" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777320 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da65f65f-be21-4a00-bb5c-f0c3f183aee1" containerName="payload-processing" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777411 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="da65f65f-be21-4a00-bb5c-f0c3f183aee1" containerName="payload-processing" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777422 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="37db937e-ba95-48cd-ac9d-d93d4c28d0cf" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777432 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c73f29d0-5265-499b-af4c-a47dec1c99dd" containerName="payload-pre-processing" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777440 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5cd9e54-98f6-42f0-8ad3-af7939680840" containerName="revoke-keys" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777448 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3900c769-1183-45c0-aa46-0cb33ddc5778" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777456 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e67049-8316-4cef-8d25-3e09e68797fe" containerName="istio-proxy" Nov 01 16:20:28.777466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.777462 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dfdff89-3026-4c1e-b2f1-3d77ace7963b" containerName="istio-proxy" Nov 01 16:20:28.784212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.782870 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.789305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.789278 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5"] Nov 01 16:20:28.800070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.800036 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:20:28.800292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.800266 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:20:28.864810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.864768 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69251bb9-549e-4499-913a-1ab14c566987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.865021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.864946 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvjws\" (UniqueName: \"kubernetes.io/projected/69251bb9-549e-4499-913a-1ab14c566987-kube-api-access-nvjws\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.865021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.865009 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69251bb9-549e-4499-913a-1ab14c566987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.966134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.966095 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvjws\" (UniqueName: \"kubernetes.io/projected/69251bb9-549e-4499-913a-1ab14c566987-kube-api-access-nvjws\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.966297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.966163 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69251bb9-549e-4499-913a-1ab14c566987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.966336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.966313 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69251bb9-549e-4499-913a-1ab14c566987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.966919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.966897 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69251bb9-549e-4499-913a-1ab14c566987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.968832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.968806 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69251bb9-549e-4499-913a-1ab14c566987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:28.973902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:28.973877 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvjws\" (UniqueName: \"kubernetes.io/projected/69251bb9-549e-4499-913a-1ab14c566987-kube-api-access-nvjws\") pod \"maas-api-revoke-sub-ce829a37-89nb5\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:29.053689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:29.053601 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:29.103679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:29.103647 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:29.421390 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:20:29.421351 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69251bb9_549e_4499_913a_1ab14c566987.slice/crio-eedc060b45f207e5b9b650fb4b6d775cfcaa9a44c9df1afd936d7a43ede0853b WatchSource:0}: Error finding container eedc060b45f207e5b9b650fb4b6d775cfcaa9a44c9df1afd936d7a43ede0853b: Status 404 returned error can't find the container with id eedc060b45f207e5b9b650fb4b6d775cfcaa9a44c9df1afd936d7a43ede0853b Nov 01 16:20:29.421844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:29.421761 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5"] Nov 01 16:20:29.978166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:29.978044 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:30.107903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:30.107859 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" event={"ID":"69251bb9-549e-4499-913a-1ab14c566987","Type":"ContainerStarted","Data":"d09b0a3cdfafa12f4216c94d9d0716b3372295157b5c7603e4106be5d7ae1dd7"} Nov 01 16:20:30.108132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:30.107963 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" event={"ID":"69251bb9-549e-4499-913a-1ab14c566987","Type":"ContainerStarted","Data":"eedc060b45f207e5b9b650fb4b6d775cfcaa9a44c9df1afd936d7a43ede0853b"} Nov 01 16:20:30.126292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:30.126234 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" podStartSLOduration=2.126220796 podStartE2EDuration="2.126220796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:30.122402988 +0000 UTC m=+1967.825568924" watchObservedRunningTime="2026-11-01 16:20:30.126220796 +0000 UTC m=+1967.829386792" Nov 01 16:20:30.735812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:30.735681 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:31.118349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:31.118314 4573 generic.go:356] "Generic (PLEG): container finished" podID="69251bb9-549e-4499-913a-1ab14c566987" containerID="d09b0a3cdfafa12f4216c94d9d0716b3372295157b5c7603e4106be5d7ae1dd7" exitCode=0 Nov 01 16:20:31.118669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:31.118375 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" event={"ID":"69251bb9-549e-4499-913a-1ab14c566987","Type":"ContainerDied","Data":"d09b0a3cdfafa12f4216c94d9d0716b3372295157b5c7603e4106be5d7ae1dd7"} Nov 01 16:20:31.484717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:31.484640 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:31.582645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:31.582610 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:31.908945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:31.908909 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:32.008723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.008683 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:32.316741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.316714 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:32.374921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.374891 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:32.427016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.426978 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69251bb9-549e-4499-913a-1ab14c566987-kube-api-access-nvjws\" (UniqueName: \"kubernetes.io/projected/69251bb9-549e-4499-913a-1ab14c566987-kube-api-access-nvjws\") pod \"69251bb9-549e-4499-913a-1ab14c566987\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " Nov 01 16:20:32.427195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.427071 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69251bb9-549e-4499-913a-1ab14c566987-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69251bb9-549e-4499-913a-1ab14c566987-maas-api-xks-ca\") pod \"69251bb9-549e-4499-913a-1ab14c566987\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " Nov 01 16:20:32.427287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.427262 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69251bb9-549e-4499-913a-1ab14c566987-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69251bb9-549e-4499-913a-1ab14c566987-maas-api-openshift-ca\") pod \"69251bb9-549e-4499-913a-1ab14c566987\" (UID: \"69251bb9-549e-4499-913a-1ab14c566987\") " Nov 01 16:20:32.427627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.427593 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69251bb9-549e-4499-913a-1ab14c566987-maas-api-openshift-ca" pod "69251bb9-549e-4499-913a-1ab14c566987" (UID: "69251bb9-549e-4499-913a-1ab14c566987"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:32.427856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.427838 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69251bb9-549e-4499-913a-1ab14c566987-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:32.429374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.429346 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69251bb9-549e-4499-913a-1ab14c566987-kube-api-access-nvjws" pod "69251bb9-549e-4499-913a-1ab14c566987" (UID: "69251bb9-549e-4499-913a-1ab14c566987"). InnerVolumeSpecName "kube-api-access-nvjws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:32.429475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.429424 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69251bb9-549e-4499-913a-1ab14c566987-maas-api-xks-ca" pod "69251bb9-549e-4499-913a-1ab14c566987" (UID: "69251bb9-549e-4499-913a-1ab14c566987"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:32.557300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.557206 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvjws\" (UniqueName: \"kubernetes.io/projected/69251bb9-549e-4499-913a-1ab14c566987-kube-api-access-nvjws\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:32.557300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.557256 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69251bb9-549e-4499-913a-1ab14c566987-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:32.611771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.609306 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:32.704783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:32.704262 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:33.137618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:33.137587 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" Nov 01 16:20:33.138046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:33.137618 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5" event={"ID":"69251bb9-549e-4499-913a-1ab14c566987","Type":"ContainerDied","Data":"eedc060b45f207e5b9b650fb4b6d775cfcaa9a44c9df1afd936d7a43ede0853b"} Nov 01 16:20:33.138046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:33.137665 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eedc060b45f207e5b9b650fb4b6d775cfcaa9a44c9df1afd936d7a43ede0853b" Nov 01 16:20:33.460846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:33.460768 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:34.144762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:34.144715 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:35.041607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:35.041574 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:35.848583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:35.848553 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:35.992811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:35.992779 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:36.102702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:36.102606 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:36.798416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:36.798356 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:37.186655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.186617 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:37.269850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.269819 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb"] Nov 01 16:20:37.270510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.270495 4573 cpu_manager.go:414] "containerMap: removing container" podUID="69251bb9-549e-4499-913a-1ab14c566987" containerName="revoke-keys" Nov 01 16:20:37.270567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.270511 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69251bb9-549e-4499-913a-1ab14c566987" containerName="revoke-keys" Nov 01 16:20:37.270633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.270623 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="69251bb9-549e-4499-913a-1ab14c566987" containerName="revoke-keys" Nov 01 16:20:37.274288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.274261 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:37.281472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.281439 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-c78897-dockercfg-ftnq4\"" Nov 01 16:20:37.281632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.281486 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-c78897\"" Nov 01 16:20:37.288115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.288080 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb"] Nov 01 16:20:37.301234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.301206 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:37.308018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.307992 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt"] Nov 01 16:20:37.319855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.319604 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt"] Nov 01 16:20:37.319855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.319718 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:37.562836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.559965 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49adbdf-60f3-4185-b408-da51610072d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb\" (UID: \"e49adbdf-60f3-4185-b408-da51610072d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:37.562836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.560472 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmv8\" (UniqueName: \"kubernetes.io/projected/e49adbdf-60f3-4185-b408-da51610072d1-kube-api-access-qqmv8\") pod \"payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb\" (UID: \"e49adbdf-60f3-4185-b408-da51610072d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:37.562836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.560526 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6297ca2d-b5a2-4e44-9e7b-cc5167735352-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt\" (UID: \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:37.562836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.561095 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswqz\" (UniqueName: \"kubernetes.io/projected/6297ca2d-b5a2-4e44-9e7b-cc5167735352-kube-api-access-xswqz\") pod \"payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt\" (UID: \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:37.662956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.662918 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswqz\" (UniqueName: \"kubernetes.io/projected/6297ca2d-b5a2-4e44-9e7b-cc5167735352-kube-api-access-xswqz\") pod \"payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt\" (UID: \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:37.663158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.663055 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49adbdf-60f3-4185-b408-da51610072d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb\" (UID: \"e49adbdf-60f3-4185-b408-da51610072d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:37.663158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.663129 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmv8\" (UniqueName: \"kubernetes.io/projected/e49adbdf-60f3-4185-b408-da51610072d1-kube-api-access-qqmv8\") pod \"payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb\" (UID: \"e49adbdf-60f3-4185-b408-da51610072d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:37.663291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.663157 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6297ca2d-b5a2-4e44-9e7b-cc5167735352-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt\" (UID: \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:37.663865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.663842 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49adbdf-60f3-4185-b408-da51610072d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb\" (UID: \"e49adbdf-60f3-4185-b408-da51610072d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:37.663911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.663879 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6297ca2d-b5a2-4e44-9e7b-cc5167735352-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt\" (UID: \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:37.672072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.672041 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmv8\" (UniqueName: \"kubernetes.io/projected/e49adbdf-60f3-4185-b408-da51610072d1-kube-api-access-qqmv8\") pod \"payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb\" (UID: \"e49adbdf-60f3-4185-b408-da51610072d1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:37.672186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.672041 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswqz\" (UniqueName: \"kubernetes.io/projected/6297ca2d-b5a2-4e44-9e7b-cc5167735352-kube-api-access-xswqz\") pod \"payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt\" (UID: \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:37.768224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.768183 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:37.890921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.890842 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:37.948380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:37.948348 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:38.179325 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:20:38.179296 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49adbdf_60f3_4185_b408_da51610072d1.slice/crio-d86aea33a95d429e72808654e935613cd2af51c6ce292f74fa377fda642d2310 WatchSource:0}: Error finding container d86aea33a95d429e72808654e935613cd2af51c6ce292f74fa377fda642d2310: Status 404 returned error can't find the container with id d86aea33a95d429e72808654e935613cd2af51c6ce292f74fa377fda642d2310 Nov 01 16:20:38.180071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:38.180049 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb"] Nov 01 16:20:38.266424 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:20:38.266402 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6297ca2d_b5a2_4e44_9e7b_cc5167735352.slice/crio-a9352585db6b376676e42d65dad887ab793ac1119e6d391b6cb5c17b7256f7e8 WatchSource:0}: Error finding container a9352585db6b376676e42d65dad887ab793ac1119e6d391b6cb5c17b7256f7e8: Status 404 returned error can't find the container with id a9352585db6b376676e42d65dad887ab793ac1119e6d391b6cb5c17b7256f7e8 Nov 01 16:20:38.268062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:38.268037 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt"] Nov 01 16:20:38.454653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:38.454568 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:39.194177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.194138 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" event={"ID":"e49adbdf-60f3-4185-b408-da51610072d1","Type":"ContainerStarted","Data":"7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768"} Nov 01 16:20:39.194383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.194195 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" event={"ID":"e49adbdf-60f3-4185-b408-da51610072d1","Type":"ContainerStarted","Data":"d86aea33a95d429e72808654e935613cd2af51c6ce292f74fa377fda642d2310"} Nov 01 16:20:39.194383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.194240 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:39.194597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.194551 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:39.195703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.195680 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:20:39.196113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.196087 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" event={"ID":"6297ca2d-b5a2-4e44-9e7b-cc5167735352","Type":"ContainerStarted","Data":"47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f"} Nov 01 16:20:39.196232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.196132 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" event={"ID":"6297ca2d-b5a2-4e44-9e7b-cc5167735352","Type":"ContainerStarted","Data":"a9352585db6b376676e42d65dad887ab793ac1119e6d391b6cb5c17b7256f7e8"} Nov 01 16:20:39.196232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.196160 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:39.219356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.219307 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" podStartSLOduration=2.219295068 podStartE2EDuration="2.219295068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:39.21578132 +0000 UTC m=+1976.918947258" watchObservedRunningTime="2026-11-01 16:20:39.219295068 +0000 UTC m=+1976.922461005" Nov 01 16:20:39.235189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:39.235143 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" podStartSLOduration=2.235131152 podStartE2EDuration="2.235131152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:39.231958367 +0000 UTC m=+1976.935124304" watchObservedRunningTime="2026-11-01 16:20:39.235131152 +0000 UTC m=+1976.938297087" Nov 01 16:20:40.152892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:40.152857 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:41.247777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:41.245117 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:41.998341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:41.998302 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:42.601617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:42.601564 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:43.482735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.482700 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:43.624772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.617007 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:43.705780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.705445 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:43.868744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.868710 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h"] Nov 01 16:20:43.874620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.874599 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:43.881821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.881803 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:20:43.885137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.885113 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:20:43.900374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.900343 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h"] Nov 01 16:20:43.920349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.920312 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:43.920543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.920523 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw2nx\" (UniqueName: \"kubernetes.io/projected/3036403e-bce1-4f73-bd3f-81b7825a32e6-kube-api-access-dw2nx\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:43.920664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:43.920647 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:44.021892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.021854 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:44.022112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.022025 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:44.022217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.022174 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw2nx\" (UniqueName: \"kubernetes.io/projected/3036403e-bce1-4f73-bd3f-81b7825a32e6-kube-api-access-dw2nx\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:44.022903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.022872 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:44.031809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.024584 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:44.032801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.032781 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw2nx\" (UniqueName: \"kubernetes.io/projected/3036403e-bce1-4f73-bd3f-81b7825a32e6-kube-api-access-dw2nx\") pod \"maas-api-revoke-keys-e2e-inf-a-04f248-9n72h\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:44.190120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.190032 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:44.479322 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:20:44.479288 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3036403e_bce1_4f73_bd3f_81b7825a32e6.slice/crio-1cf54e00e6f3091c024aa8971ec51bf7e663e8d1755d06d87431910b2c079dc8 WatchSource:0}: Error finding container 1cf54e00e6f3091c024aa8971ec51bf7e663e8d1755d06d87431910b2c079dc8: Status 404 returned error can't find the container with id 1cf54e00e6f3091c024aa8971ec51bf7e663e8d1755d06d87431910b2c079dc8 Nov 01 16:20:44.479891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.479861 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h"] Nov 01 16:20:44.751984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:44.751894 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:45.299157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:45.299129 4573 generic.go:356] "Generic (PLEG): container finished" podID="3036403e-bce1-4f73-bd3f-81b7825a32e6" containerID="a31f0699772a6b8cbb6df10017c32c688954a850410b24dc52b9ca03442833d3" exitCode=0 Nov 01 16:20:45.299290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:45.299211 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" event={"ID":"3036403e-bce1-4f73-bd3f-81b7825a32e6","Type":"ContainerDied","Data":"a31f0699772a6b8cbb6df10017c32c688954a850410b24dc52b9ca03442833d3"} Nov 01 16:20:45.299290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:45.299268 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" event={"ID":"3036403e-bce1-4f73-bd3f-81b7825a32e6","Type":"ContainerStarted","Data":"1cf54e00e6f3091c024aa8971ec51bf7e663e8d1755d06d87431910b2c079dc8"} Nov 01 16:20:45.529791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:45.529677 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:46.457774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.448655 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:46.532723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.532699 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:46.565513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.565441 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-openshift-ca\") pod \"3036403e-bce1-4f73-bd3f-81b7825a32e6\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " Nov 01 16:20:46.565697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.565677 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3036403e-bce1-4f73-bd3f-81b7825a32e6-kube-api-access-dw2nx\" (UniqueName: \"kubernetes.io/projected/3036403e-bce1-4f73-bd3f-81b7825a32e6-kube-api-access-dw2nx\") pod \"3036403e-bce1-4f73-bd3f-81b7825a32e6\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " Nov 01 16:20:46.565819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.565796 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-xks-ca\") pod \"3036403e-bce1-4f73-bd3f-81b7825a32e6\" (UID: \"3036403e-bce1-4f73-bd3f-81b7825a32e6\") " Nov 01 16:20:46.565913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.565886 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-openshift-ca" pod "3036403e-bce1-4f73-bd3f-81b7825a32e6" (UID: "3036403e-bce1-4f73-bd3f-81b7825a32e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:20:46.567667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.566684 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:46.569327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.569301 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-xks-ca" pod "3036403e-bce1-4f73-bd3f-81b7825a32e6" (UID: "3036403e-bce1-4f73-bd3f-81b7825a32e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:20:46.575956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.575927 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3036403e-bce1-4f73-bd3f-81b7825a32e6-kube-api-access-dw2nx" pod "3036403e-bce1-4f73-bd3f-81b7825a32e6" (UID: "3036403e-bce1-4f73-bd3f-81b7825a32e6"). InnerVolumeSpecName "kube-api-access-dw2nx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:20:46.668242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.668212 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3036403e-bce1-4f73-bd3f-81b7825a32e6-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:46.668242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:46.668241 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw2nx\" (UniqueName: \"kubernetes.io/projected/3036403e-bce1-4f73-bd3f-81b7825a32e6-kube-api-access-dw2nx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:20:47.026235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:47.026202 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:47.318143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:47.318051 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" Nov 01 16:20:47.318143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:47.318074 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h" event={"ID":"3036403e-bce1-4f73-bd3f-81b7825a32e6","Type":"ContainerDied","Data":"1cf54e00e6f3091c024aa8971ec51bf7e663e8d1755d06d87431910b2c079dc8"} Nov 01 16:20:47.318143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:47.318130 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cf54e00e6f3091c024aa8971ec51bf7e663e8d1755d06d87431910b2c079dc8" Nov 01 16:20:47.767961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:47.767931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:48.565303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:48.565273 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:49.362782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:49.358886 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:50.045618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:50.045585 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:50.208682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:50.208653 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:20:50.208875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:50.208722 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:20:50.781032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:50.781002 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:51.652555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:51.652521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:52.452722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:52.452679 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:53.152139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:53.152103 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:53.960861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:53.960823 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:54.060789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.060718 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:54.371493 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:20:54.371456 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:20:54.622189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.622107 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr"] Nov 01 16:20:54.622852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.622830 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3036403e-bce1-4f73-bd3f-81b7825a32e6" containerName="revoke-keys" Nov 01 16:20:54.622852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.622850 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3036403e-bce1-4f73-bd3f-81b7825a32e6" containerName="revoke-keys" Nov 01 16:20:54.623133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.623117 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3036403e-bce1-4f73-bd3f-81b7825a32e6" containerName="revoke-keys" Nov 01 16:20:54.649351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.649320 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr"] Nov 01 16:20:54.649515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.649446 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.652968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.652937 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c78897-serving-cert\"" Nov 01 16:20:54.663288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.663252 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:54.700410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.700384 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgck\" (UniqueName: \"kubernetes.io/projected/80b38e9a-4645-4a5b-877d-a3228d10f008-kube-api-access-hdgck\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.700632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.700597 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.700710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.700632 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b38e9a-4645-4a5b-877d-a3228d10f008-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.700837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.700813 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.763813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.763767 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:54.815793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.807242 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.815793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.807394 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgck\" (UniqueName: \"kubernetes.io/projected/80b38e9a-4645-4a5b-877d-a3228d10f008-kube-api-access-hdgck\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.815793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.807598 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.815793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.807631 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b38e9a-4645-4a5b-877d-a3228d10f008-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.815793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.808602 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b38e9a-4645-4a5b-877d-a3228d10f008-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.823792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.820568 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.823792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.821342 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.827783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.825779 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgck\" (UniqueName: \"kubernetes.io/projected/80b38e9a-4645-4a5b-877d-a3228d10f008-kube-api-access-hdgck\") pod \"maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:54.865678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.865626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:54.968880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:54.968792 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:55.293531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:55.293499 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr"] Nov 01 16:20:55.295431 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:20:55.295388 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b38e9a_4645_4a5b_877d_a3228d10f008.slice/crio-b0414faf40fd8e7de5a0a7899d3a71aca31cf0be6a6455b8c1c227769ced2d2d WatchSource:0}: Error finding container b0414faf40fd8e7de5a0a7899d3a71aca31cf0be6a6455b8c1c227769ced2d2d: Status 404 returned error can't find the container with id b0414faf40fd8e7de5a0a7899d3a71aca31cf0be6a6455b8c1c227769ced2d2d Nov 01 16:20:55.296797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:55.296775 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:20:55.389068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:55.389022 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" event={"ID":"80b38e9a-4645-4a5b-877d-a3228d10f008","Type":"ContainerStarted","Data":"b0414faf40fd8e7de5a0a7899d3a71aca31cf0be6a6455b8c1c227769ced2d2d"} Nov 01 16:20:55.568602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:55.568520 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:56.398971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:56.398939 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" event={"ID":"80b38e9a-4645-4a5b-877d-a3228d10f008","Type":"ContainerStarted","Data":"b81d6596e7b563c148810aa20d341a3efb7d51da8c55fe2fccbe9b1e9a3e0b56"} Nov 01 16:20:56.399420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:56.399076 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:20:56.418113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:56.418055 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" podStartSLOduration=1.972336587 podStartE2EDuration="2.418041806s" totalImagesPullingTime="445.705219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:20:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:20:56.414996989 +0000 UTC m=+1994.118162925" watchObservedRunningTime="2026-11-01 16:20:56.418041806 +0000 UTC m=+1994.121207741" Nov 01 16:20:56.429466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:56.429418 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:56.653331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:56.653250 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:57.351189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:57.351154 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:58.174050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:58.174019 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:59.022231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:59.022201 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:20:59.692260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:20:59.692209 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:00.461866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:00.461818 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:01.169796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:01.169760 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:01.857744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:01.857713 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:02.415740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:02.415711 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:21:02.558186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:02.558138 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:03.246766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:03.246725 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:03.878658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:03.878627 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:04.685363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:04.685325 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:05.229272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:05.229226 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:05.374735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:05.374696 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:05.452780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:05.450224 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:05.657057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:05.657012 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h"] Nov 01 16:21:05.662057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:05.662030 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-04f248-9n72h"] Nov 01 16:21:05.727685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:05.727650 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:06.103829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:06.103719 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:06.459639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:06.459608 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3036403e-bce1-4f73-bd3f-81b7825a32e6" path="/var/lib/kubelet/pods/3036403e-bce1-4f73-bd3f-81b7825a32e6/volumes" Nov 01 16:21:06.472617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:06.472575 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:07.160634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.160602 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:07.485278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.485202 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn"] Nov 01 16:21:07.489736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.489719 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.496641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.496615 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn"] Nov 01 16:21:07.496966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.496944 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:21:07.498159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.498137 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:21:07.584502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.584472 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.584675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.584587 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.584675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.584664 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh8x8\" (UniqueName: \"kubernetes.io/projected/19a0a294-995d-462c-aa08-0c51bb869ae1-kube-api-access-vh8x8\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.685606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.685571 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.685829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.685648 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.685829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.685705 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh8x8\" (UniqueName: \"kubernetes.io/projected/19a0a294-995d-462c-aa08-0c51bb869ae1-kube-api-access-vh8x8\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.686376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.686357 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.687929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.687900 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.693385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.693364 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh8x8\" (UniqueName: \"kubernetes.io/projected/19a0a294-995d-462c-aa08-0c51bb869ae1-kube-api-access-vh8x8\") pod \"maas-api-revoke-sub-e045478a-8r8kn\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.810608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.810513 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:07.875255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:07.875109 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:08.097822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:08.097791 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn"] Nov 01 16:21:08.098508 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:21:08.098483 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a0a294_995d_462c_aa08_0c51bb869ae1.slice/crio-3b505efed01c0d4ce7ee6d80ebbd537045616561b6b298acda8dd732f47faba3 WatchSource:0}: Error finding container 3b505efed01c0d4ce7ee6d80ebbd537045616561b6b298acda8dd732f47faba3: Status 404 returned error can't find the container with id 3b505efed01c0d4ce7ee6d80ebbd537045616561b6b298acda8dd732f47faba3 Nov 01 16:21:08.541246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:08.541205 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" event={"ID":"19a0a294-995d-462c-aa08-0c51bb869ae1","Type":"ContainerStarted","Data":"cb535016728b200ede6fde54538e3c4cc5a20cf19d575532fa5450cd3591d5e8"} Nov 01 16:21:08.541667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:08.541273 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" event={"ID":"19a0a294-995d-462c-aa08-0c51bb869ae1","Type":"ContainerStarted","Data":"3b505efed01c0d4ce7ee6d80ebbd537045616561b6b298acda8dd732f47faba3"} Nov 01 16:21:08.554831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:08.554792 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:08.563990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:08.563951 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" podStartSLOduration=1.563939355 podStartE2EDuration="1.563939355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:08.559335485 +0000 UTC m=+2006.262501423" watchObservedRunningTime="2026-11-01 16:21:08.563939355 +0000 UTC m=+2006.267105291" Nov 01 16:21:09.391101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:09.391072 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:09.550670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:09.550639 4573 generic.go:356] "Generic (PLEG): container finished" podID="19a0a294-995d-462c-aa08-0c51bb869ae1" containerID="cb535016728b200ede6fde54538e3c4cc5a20cf19d575532fa5450cd3591d5e8" exitCode=0 Nov 01 16:21:09.551096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:09.550681 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" event={"ID":"19a0a294-995d-462c-aa08-0c51bb869ae1","Type":"ContainerDied","Data":"cb535016728b200ede6fde54538e3c4cc5a20cf19d575532fa5450cd3591d5e8"} Nov 01 16:21:10.083705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.083669 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:10.735059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.735029 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:10.762029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.761999 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:10.849418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.849340 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a0a294-995d-462c-aa08-0c51bb869ae1-kube-api-access-vh8x8\" (UniqueName: \"kubernetes.io/projected/19a0a294-995d-462c-aa08-0c51bb869ae1-kube-api-access-vh8x8\") pod \"19a0a294-995d-462c-aa08-0c51bb869ae1\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " Nov 01 16:21:10.849585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.849440 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-xks-ca\") pod \"19a0a294-995d-462c-aa08-0c51bb869ae1\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " Nov 01 16:21:10.849585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.849557 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-openshift-ca\") pod \"19a0a294-995d-462c-aa08-0c51bb869ae1\" (UID: \"19a0a294-995d-462c-aa08-0c51bb869ae1\") " Nov 01 16:21:10.849997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.849951 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-openshift-ca" pod "19a0a294-995d-462c-aa08-0c51bb869ae1" (UID: "19a0a294-995d-462c-aa08-0c51bb869ae1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:10.850266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.850234 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:10.851621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.851592 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-xks-ca" pod "19a0a294-995d-462c-aa08-0c51bb869ae1" (UID: "19a0a294-995d-462c-aa08-0c51bb869ae1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:10.851878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.851853 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a0a294-995d-462c-aa08-0c51bb869ae1-kube-api-access-vh8x8" pod "19a0a294-995d-462c-aa08-0c51bb869ae1" (UID: "19a0a294-995d-462c-aa08-0c51bb869ae1"). InnerVolumeSpecName "kube-api-access-vh8x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:10.951711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.951676 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh8x8\" (UniqueName: \"kubernetes.io/projected/19a0a294-995d-462c-aa08-0c51bb869ae1-kube-api-access-vh8x8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:10.951711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:10.951709 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a0a294-995d-462c-aa08-0c51bb869ae1-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:11.385583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:11.385542 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:11.569568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:11.569542 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" Nov 01 16:21:11.569745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:11.569569 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn" event={"ID":"19a0a294-995d-462c-aa08-0c51bb869ae1","Type":"ContainerDied","Data":"3b505efed01c0d4ce7ee6d80ebbd537045616561b6b298acda8dd732f47faba3"} Nov 01 16:21:11.569745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:11.569613 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b505efed01c0d4ce7ee6d80ebbd537045616561b6b298acda8dd732f47faba3" Nov 01 16:21:12.254033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:12.253976 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:12.510389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:12.510253 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:12.592625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:12.592571 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:12.810504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:12.810399 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:13.011541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.011484 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:13.371091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.371041 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:13.374862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.374840 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4"] Nov 01 16:21:13.375677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.375658 4573 cpu_manager.go:414] "containerMap: removing container" podUID="19a0a294-995d-462c-aa08-0c51bb869ae1" containerName="revoke-keys" Nov 01 16:21:13.375677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.375676 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a0a294-995d-462c-aa08-0c51bb869ae1" containerName="revoke-keys" Nov 01 16:21:13.375890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.375852 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a0a294-995d-462c-aa08-0c51bb869ae1" containerName="revoke-keys" Nov 01 16:21:13.379961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.379941 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.390198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.390170 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4"] Nov 01 16:21:13.396584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.396559 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:21:13.396699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.396612 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:21:13.435316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.435228 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.435554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.435334 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.435554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.435421 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lgsb\" (UniqueName: \"kubernetes.io/projected/c187c622-673a-47f4-b3e6-c6d3c763049e-kube-api-access-9lgsb\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.537165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.537123 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.537344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.537231 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lgsb\" (UniqueName: \"kubernetes.io/projected/c187c622-673a-47f4-b3e6-c6d3c763049e-kube-api-access-9lgsb\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.537506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.537489 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.537970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.537930 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.540159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.540134 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.545476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.545451 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lgsb\" (UniqueName: \"kubernetes.io/projected/c187c622-673a-47f4-b3e6-c6d3c763049e-kube-api-access-9lgsb\") pod \"maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.702394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.702313 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:13.987907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:13.987877 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4"] Nov 01 16:21:13.988021 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:21:13.987904 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc187c622_673a_47f4_b3e6_c6d3c763049e.slice/crio-ab85a732f2ac21a6195356beb6a14bff90ed5afa1dc2443bd1c495ca3763b97a WatchSource:0}: Error finding container ab85a732f2ac21a6195356beb6a14bff90ed5afa1dc2443bd1c495ca3763b97a: Status 404 returned error can't find the container with id ab85a732f2ac21a6195356beb6a14bff90ed5afa1dc2443bd1c495ca3763b97a Nov 01 16:21:14.296721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:14.296636 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:14.606189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:14.606102 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" event={"ID":"c187c622-673a-47f4-b3e6-c6d3c763049e","Type":"ContainerStarted","Data":"6813356da2972d9b618f4589b6a2cbc3278200a6331a8f2ba795d60b375f19eb"} Nov 01 16:21:14.606189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:14.606162 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" event={"ID":"c187c622-673a-47f4-b3e6-c6d3c763049e","Type":"ContainerStarted","Data":"ab85a732f2ac21a6195356beb6a14bff90ed5afa1dc2443bd1c495ca3763b97a"} Nov 01 16:21:14.624841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:14.624791 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" podStartSLOduration=1.624741186 podStartE2EDuration="1.624741186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:14.622678652 +0000 UTC m=+2012.325844584" watchObservedRunningTime="2026-11-01 16:21:14.624741186 +0000 UTC m=+2012.327907121" Nov 01 16:21:15.109377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:15.109325 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:15.617340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:15.617303 4573 generic.go:356] "Generic (PLEG): container finished" podID="c187c622-673a-47f4-b3e6-c6d3c763049e" containerID="6813356da2972d9b618f4589b6a2cbc3278200a6331a8f2ba795d60b375f19eb" exitCode=0 Nov 01 16:21:15.617772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:15.617343 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" event={"ID":"c187c622-673a-47f4-b3e6-c6d3c763049e","Type":"ContainerDied","Data":"6813356da2972d9b618f4589b6a2cbc3278200a6331a8f2ba795d60b375f19eb"} Nov 01 16:21:15.974479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:15.974396 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:16.478953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:16.478914 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:16.699577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:16.699542 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:16.880199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:16.880176 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:17.021998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.021959 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-xks-ca\") pod \"c187c622-673a-47f4-b3e6-c6d3c763049e\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " Nov 01 16:21:17.022198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.022031 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-openshift-ca\") pod \"c187c622-673a-47f4-b3e6-c6d3c763049e\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " Nov 01 16:21:17.022198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.022067 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c187c622-673a-47f4-b3e6-c6d3c763049e-kube-api-access-9lgsb\" (UniqueName: \"kubernetes.io/projected/c187c622-673a-47f4-b3e6-c6d3c763049e-kube-api-access-9lgsb\") pod \"c187c622-673a-47f4-b3e6-c6d3c763049e\" (UID: \"c187c622-673a-47f4-b3e6-c6d3c763049e\") " Nov 01 16:21:17.022380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.022357 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-openshift-ca" pod "c187c622-673a-47f4-b3e6-c6d3c763049e" (UID: "c187c622-673a-47f4-b3e6-c6d3c763049e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:17.022641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.022627 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:17.024153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.024122 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-xks-ca" pod "c187c622-673a-47f4-b3e6-c6d3c763049e" (UID: "c187c622-673a-47f4-b3e6-c6d3c763049e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:17.024264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.024196 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c187c622-673a-47f4-b3e6-c6d3c763049e-kube-api-access-9lgsb" pod "c187c622-673a-47f4-b3e6-c6d3c763049e" (UID: "c187c622-673a-47f4-b3e6-c6d3c763049e"). InnerVolumeSpecName "kube-api-access-9lgsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:17.125020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.124989 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187c622-673a-47f4-b3e6-c6d3c763049e-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:17.125020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.125020 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lgsb\" (UniqueName: \"kubernetes.io/projected/c187c622-673a-47f4-b3e6-c6d3c763049e-kube-api-access-9lgsb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:17.511068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.510974 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:17.702398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.702357 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" event={"ID":"c187c622-673a-47f4-b3e6-c6d3c763049e","Type":"ContainerDied","Data":"ab85a732f2ac21a6195356beb6a14bff90ed5afa1dc2443bd1c495ca3763b97a"} Nov 01 16:21:17.702933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.702419 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab85a732f2ac21a6195356beb6a14bff90ed5afa1dc2443bd1c495ca3763b97a" Nov 01 16:21:17.702933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:17.702493 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4" Nov 01 16:21:18.285602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:18.285554 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:19.060903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:19.060870 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:19.687416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:19.687386 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:20.477109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:20.477073 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:20.854153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:20.854068 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:21.369945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:21.369913 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:22.039158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:22.039120 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:22.875831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:22.875790 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:23.555141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:23.555109 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:23.817859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:23.790301 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:24.012054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:24.012021 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:24.365471 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:21:24.365432 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:21:24.651616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:24.651531 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:24.973811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:24.973728 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:25.614930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:25.614883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:25.751864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:25.751812 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:26.023019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.022922 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r"] Nov 01 16:21:26.023841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.023811 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c187c622-673a-47f4-b3e6-c6d3c763049e" containerName="revoke-keys" Nov 01 16:21:26.023841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.023837 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c187c622-673a-47f4-b3e6-c6d3c763049e" containerName="revoke-keys" Nov 01 16:21:26.024105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.024088 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c187c622-673a-47f4-b3e6-c6d3c763049e" containerName="revoke-keys" Nov 01 16:21:26.028280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.028256 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.031916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.031892 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:21:26.032191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.032174 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:21:26.036198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.036170 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r"] Nov 01 16:21:26.122590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.122563 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjpg\" (UniqueName: \"kubernetes.io/projected/62e9e631-16f2-49f0-92ac-4596545b643c-kube-api-access-dxjpg\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.122791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.122633 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.122858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.122805 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.224035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.224008 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjpg\" (UniqueName: \"kubernetes.io/projected/62e9e631-16f2-49f0-92ac-4596545b643c-kube-api-access-dxjpg\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.224193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.224080 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.224266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.224235 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.225028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.225004 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.226801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.226775 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.233692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.233666 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjpg\" (UniqueName: \"kubernetes.io/projected/62e9e631-16f2-49f0-92ac-4596545b643c-kube-api-access-dxjpg\") pod \"maas-api-revoke-sub-c6c25547-np89r\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.361278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.361254 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:26.467739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.467701 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:26.651147 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:21:26.651111 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e9e631_16f2_49f0_92ac_4596545b643c.slice/crio-9952d9471a8ffc36aa088a38dda3b9c15c42386e2800baf57b3cde447fa77fa0 WatchSource:0}: Error finding container 9952d9471a8ffc36aa088a38dda3b9c15c42386e2800baf57b3cde447fa77fa0: Status 404 returned error can't find the container with id 9952d9471a8ffc36aa088a38dda3b9c15c42386e2800baf57b3cde447fa77fa0 Nov 01 16:21:26.651553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.651534 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r"] Nov 01 16:21:26.827719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:26.827684 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" event={"ID":"62e9e631-16f2-49f0-92ac-4596545b643c","Type":"ContainerStarted","Data":"9952d9471a8ffc36aa088a38dda3b9c15c42386e2800baf57b3cde447fa77fa0"} Nov 01 16:21:27.210088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:27.210052 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:27.837137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:27.837098 4573 generic.go:356] "Generic (PLEG): container finished" podID="62e9e631-16f2-49f0-92ac-4596545b643c" containerID="8117dfb52051936dcda31922aacc739b0ff3371e005d6c7d0101517d9cea8286" exitCode=0 Nov 01 16:21:27.837548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:27.837139 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" event={"ID":"62e9e631-16f2-49f0-92ac-4596545b643c","Type":"ContainerDied","Data":"8117dfb52051936dcda31922aacc739b0ff3371e005d6c7d0101517d9cea8286"} Nov 01 16:21:27.978903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:27.978867 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:28.865189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:28.865151 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:29.071198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.071169 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:29.184693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.184663 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62e9e631-16f2-49f0-92ac-4596545b643c-kube-api-access-dxjpg\" (UniqueName: \"kubernetes.io/projected/62e9e631-16f2-49f0-92ac-4596545b643c-kube-api-access-dxjpg\") pod \"62e9e631-16f2-49f0-92ac-4596545b643c\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " Nov 01 16:21:29.184693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.184697 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-xks-ca\") pod \"62e9e631-16f2-49f0-92ac-4596545b643c\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " Nov 01 16:21:29.185024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.184742 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-openshift-ca\") pod \"62e9e631-16f2-49f0-92ac-4596545b643c\" (UID: \"62e9e631-16f2-49f0-92ac-4596545b643c\") " Nov 01 16:21:29.185284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.185254 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-openshift-ca" pod "62e9e631-16f2-49f0-92ac-4596545b643c" (UID: "62e9e631-16f2-49f0-92ac-4596545b643c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:29.185592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.185562 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.187253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.187234 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-xks-ca" pod "62e9e631-16f2-49f0-92ac-4596545b643c" (UID: "62e9e631-16f2-49f0-92ac-4596545b643c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:29.188523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.188500 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e9e631-16f2-49f0-92ac-4596545b643c-kube-api-access-dxjpg" pod "62e9e631-16f2-49f0-92ac-4596545b643c" (UID: "62e9e631-16f2-49f0-92ac-4596545b643c"). InnerVolumeSpecName "kube-api-access-dxjpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:29.287149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.287117 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjpg\" (UniqueName: \"kubernetes.io/projected/62e9e631-16f2-49f0-92ac-4596545b643c-kube-api-access-dxjpg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.287149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.287147 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62e9e631-16f2-49f0-92ac-4596545b643c-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:29.554664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.554582 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47"] Nov 01 16:21:29.554939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.554858 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" podUID="d07785d5-56d1-4261-b44e-3397f98d9cc9" containerName="maas-api" containerID="cri-o://d59ea225ac276b06bba7850b31446cde3f42f16b30f98fda5bfd9002c2536784" gracePeriod=30 Nov 01 16:21:29.594549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.594522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:29.725871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.725835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:29.856081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.854429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:29.955772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.954817 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" event={"ID":"62e9e631-16f2-49f0-92ac-4596545b643c","Type":"ContainerDied","Data":"9952d9471a8ffc36aa088a38dda3b9c15c42386e2800baf57b3cde447fa77fa0"} Nov 01 16:21:29.955772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.954884 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9952d9471a8ffc36aa088a38dda3b9c15c42386e2800baf57b3cde447fa77fa0" Nov 01 16:21:29.955772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.954967 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r" Nov 01 16:21:29.960374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.959216 4573 generic.go:356] "Generic (PLEG): container finished" podID="d07785d5-56d1-4261-b44e-3397f98d9cc9" containerID="d59ea225ac276b06bba7850b31446cde3f42f16b30f98fda5bfd9002c2536784" exitCode=0 Nov 01 16:21:29.960374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:29.959353 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" event={"ID":"d07785d5-56d1-4261-b44e-3397f98d9cc9","Type":"ContainerDied","Data":"d59ea225ac276b06bba7850b31446cde3f42f16b30f98fda5bfd9002c2536784"} Nov 01 16:21:30.011192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.011125 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4"] Nov 01 16:21:30.017376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.017310 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-10b00a-ss9r4"] Nov 01 16:21:30.059117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.059083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:30.092563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.092541 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:21:30.244140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.244109 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-maas-api-tls\") pod \"d07785d5-56d1-4261-b44e-3397f98d9cc9\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " Nov 01 16:21:30.244337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.244284 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-metrics-certs\") pod \"d07785d5-56d1-4261-b44e-3397f98d9cc9\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " Nov 01 16:21:30.244408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.244370 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07785d5-56d1-4261-b44e-3397f98d9cc9-kube-api-access-24z6j\" (UniqueName: \"kubernetes.io/projected/d07785d5-56d1-4261-b44e-3397f98d9cc9-kube-api-access-24z6j\") pod \"d07785d5-56d1-4261-b44e-3397f98d9cc9\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " Nov 01 16:21:30.244735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.244470 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07785d5-56d1-4261-b44e-3397f98d9cc9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07785d5-56d1-4261-b44e-3397f98d9cc9-openshift-service-ca\") pod \"d07785d5-56d1-4261-b44e-3397f98d9cc9\" (UID: \"d07785d5-56d1-4261-b44e-3397f98d9cc9\") " Nov 01 16:21:30.245693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.245400 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07785d5-56d1-4261-b44e-3397f98d9cc9-openshift-service-ca" pod "d07785d5-56d1-4261-b44e-3397f98d9cc9" (UID: "d07785d5-56d1-4261-b44e-3397f98d9cc9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:30.247573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.247545 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d07785d5-56d1-4261-b44e-3397f98d9cc9-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:30.248853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.248815 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07785d5-56d1-4261-b44e-3397f98d9cc9-kube-api-access-24z6j" pod "d07785d5-56d1-4261-b44e-3397f98d9cc9" (UID: "d07785d5-56d1-4261-b44e-3397f98d9cc9"). InnerVolumeSpecName "kube-api-access-24z6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:30.250626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.250598 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-maas-api-tls" pod "d07785d5-56d1-4261-b44e-3397f98d9cc9" (UID: "d07785d5-56d1-4261-b44e-3397f98d9cc9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:30.283579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.283537 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-metrics-certs" pod "d07785d5-56d1-4261-b44e-3397f98d9cc9" (UID: "d07785d5-56d1-4261-b44e-3397f98d9cc9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:30.327916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.327882 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:30.349482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.349452 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:30.349482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.349481 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24z6j\" (UniqueName: \"kubernetes.io/projected/d07785d5-56d1-4261-b44e-3397f98d9cc9-kube-api-access-24z6j\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:30.349681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.349492 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d07785d5-56d1-4261-b44e-3397f98d9cc9-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:30.418917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.418883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:30.514832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.511843 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c187c622-673a-47f4-b3e6-c6d3c763049e" path="/var/lib/kubelet/pods/c187c622-673a-47f4-b3e6-c6d3c763049e/volumes" Nov 01 16:21:30.969129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.969101 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" Nov 01 16:21:30.969594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.969125 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47" event={"ID":"d07785d5-56d1-4261-b44e-3397f98d9cc9","Type":"ContainerDied","Data":"3aede44e1922dabc50a968a9cc35f8757ae7841f86b761c8e61b9b71338a6109"} Nov 01 16:21:30.969594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.969201 4573 scope.go:122] "RemoveContainer" containerID="d59ea225ac276b06bba7850b31446cde3f42f16b30f98fda5bfd9002c2536784" Nov 01 16:21:30.988784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.988744 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47"] Nov 01 16:21:30.994388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:30.994365 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-10b00a-695cd4f675-r2j47"] Nov 01 16:21:31.113622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:31.113592 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:31.831739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:31.831705 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:31.951780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:31.942672 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:32.459478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:32.459448 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07785d5-56d1-4261-b44e-3397f98d9cc9" path="/var/lib/kubelet/pods/d07785d5-56d1-4261-b44e-3397f98d9cc9/volumes" Nov 01 16:21:32.609557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:32.609523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:33.278341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:33.278307 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:33.952881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:33.952845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:34.651397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:34.651362 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:35.665981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:35.665934 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:36.996006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:36.984198 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:37.882369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:37.882336 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:37.996172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:37.996131 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:38.212505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:38.212421 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z"] Nov 01 16:21:38.212697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:38.212664 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" containerID="cri-o://262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad" gracePeriod=30 Nov 01 16:21:38.242194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:38.242162 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:38.992418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:38.992383 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:39.100369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:39.100329 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:39.450095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:39.450060 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:39.692104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:39.692054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:40.169235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:40.169196 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:41.074743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:41.074716 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:41.747727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:41.747696 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:42.666489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.666452 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:42.924101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.924023 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l"] Nov 01 16:21:42.924947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.924926 4573 cpu_manager.go:414] "containerMap: removing container" podUID="62e9e631-16f2-49f0-92ac-4596545b643c" containerName="revoke-keys" Nov 01 16:21:42.924947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.924947 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62e9e631-16f2-49f0-92ac-4596545b643c" containerName="revoke-keys" Nov 01 16:21:42.925127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.925001 4573 cpu_manager.go:414] "containerMap: removing container" podUID="d07785d5-56d1-4261-b44e-3397f98d9cc9" containerName="maas-api" Nov 01 16:21:42.925127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.925011 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07785d5-56d1-4261-b44e-3397f98d9cc9" containerName="maas-api" Nov 01 16:21:42.925231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.925205 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07785d5-56d1-4261-b44e-3397f98d9cc9" containerName="maas-api" Nov 01 16:21:42.925289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.925230 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="62e9e631-16f2-49f0-92ac-4596545b643c" containerName="revoke-keys" Nov 01 16:21:42.928427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.928401 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:42.929400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.929383 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:42.933669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.933645 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:21:42.934105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.933718 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:21:42.936809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:42.936780 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l"] Nov 01 16:21:43.057732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.057697 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.057969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.057921 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdpwb\" (UniqueName: \"kubernetes.io/projected/1ed0c712-d79d-4c4c-b3ce-7242c785153c-kube-api-access-kdpwb\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.057969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.057955 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.159972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.159935 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdpwb\" (UniqueName: \"kubernetes.io/projected/1ed0c712-d79d-4c4c-b3ce-7242c785153c-kube-api-access-kdpwb\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.159972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.159975 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.160197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.160122 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.160823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.160798 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.162402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.162374 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.167489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.167463 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdpwb\" (UniqueName: \"kubernetes.io/projected/1ed0c712-d79d-4c4c-b3ce-7242c785153c-kube-api-access-kdpwb\") pod \"maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.231460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.231428 4573 patch_prober.go:29] interesting pod/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" start-of-body= Nov 01 16:21:43.231599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.231501 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.185:15021/healthz/ready\": dial tcp 10.133.0.185:15021: connect: connection refused" Nov 01 16:21:43.256632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.256608 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:43.414332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.414174 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:43.504608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.504523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:43.544355 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:21:43.544331 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ed0c712_d79d_4c4c_b3ce_7242c785153c.slice/crio-1582631eb5316924ed73b1095d38e4b3d55b829869f2964a2bb6538fa2f5b478 WatchSource:0}: Error finding container 1582631eb5316924ed73b1095d38e4b3d55b829869f2964a2bb6538fa2f5b478: Status 404 returned error can't find the container with id 1582631eb5316924ed73b1095d38e4b3d55b829869f2964a2bb6538fa2f5b478 Nov 01 16:21:43.545007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.544993 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:21:43.545200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.545176 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l"] Nov 01 16:21:43.672163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672124 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-token\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-token\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.672349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672187 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-credential-socket\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.672349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672212 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-data\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.672349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672237 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istiod-ca-cert\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.672349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672264 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-envoy\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.672349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672325 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-socket\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.672680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672401 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-certs\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.672680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672447 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-podinfo\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.672846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.672790 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-kube-api-access-8sxzx\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-kube-api-access-8sxzx\") pod \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\" (UID: \"9c921e79-5128-4b20-bdc3-302e60b3e5d6\") " Nov 01 16:21:43.673094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.673069 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-certs" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.673728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.673542 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-socket" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.673728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.673556 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-credential-socket" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.674053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.674012 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-data" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.674321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.674283 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istiod-ca-cert" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:43.674693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.674611 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.674693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.674639 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.674693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.674655 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.674693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.674668 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.675541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.675516 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-token" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:43.675718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.675669 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-kube-api-access-8sxzx" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "kube-api-access-8sxzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:43.676045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.676021 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-podinfo" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:21:43.677880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.677858 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-envoy" pod "9c921e79-5128-4b20-bdc3-302e60b3e5d6" (UID: "9c921e79-5128-4b20-bdc3-302e60b3e5d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:21:43.776448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.776411 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sxzx\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-kube-api-access-8sxzx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.776622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.776450 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.776622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.776467 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.776622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.776481 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.776622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.776491 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c921e79-5128-4b20-bdc3-302e60b3e5d6-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:43.999633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:43.999603 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:44.122088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.122046 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" event={"ID":"1ed0c712-d79d-4c4c-b3ce-7242c785153c","Type":"ContainerStarted","Data":"0e9dcf30619159407c9bb625e0432cc8c7774a4aedb5679b5db70d6f5b8931d6"} Nov 01 16:21:44.122277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.122115 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" event={"ID":"1ed0c712-d79d-4c4c-b3ce-7242c785153c","Type":"ContainerStarted","Data":"1582631eb5316924ed73b1095d38e4b3d55b829869f2964a2bb6538fa2f5b478"} Nov 01 16:21:44.123914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.123884 4573 generic.go:356] "Generic (PLEG): container finished" podID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerID="262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad" exitCode=0 Nov 01 16:21:44.124047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.123934 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" Nov 01 16:21:44.124047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.123970 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" event={"ID":"9c921e79-5128-4b20-bdc3-302e60b3e5d6","Type":"ContainerDied","Data":"262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad"} Nov 01 16:21:44.124047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.124033 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z" event={"ID":"9c921e79-5128-4b20-bdc3-302e60b3e5d6","Type":"ContainerDied","Data":"dfac2842dd817e8e59b807e2b139ac04c35277d3c799079d16f24bb36a831b4f"} Nov 01 16:21:44.124227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.124065 4573 scope.go:122] "RemoveContainer" containerID="262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad" Nov 01 16:21:44.135895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.135877 4573 scope.go:122] "RemoveContainer" containerID="6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d" Nov 01 16:21:44.143171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.143121 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" podStartSLOduration=2.143103456 podStartE2EDuration="2.143103456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:44.138435186 +0000 UTC m=+2041.841601121" watchObservedRunningTime="2026-11-01 16:21:44.143103456 +0000 UTC m=+2041.846269393" Nov 01 16:21:44.148279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.148262 4573 scope.go:122] "RemoveContainer" containerID="262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad" Nov 01 16:21:44.148556 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:21:44.148532 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad\": container with ID starting with 262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad not found: ID does not exist" containerID="262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad" Nov 01 16:21:44.148648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.148569 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad"} err="failed to get container status \"262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad\": rpc error: code = NotFound desc = could not find container \"262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad\": container with ID starting with 262d42913340a5c987dc76628972f49feae1b414dd0d1ea0b5ed6465942313ad not found: ID does not exist" Nov 01 16:21:44.148648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.148594 4573 scope.go:122] "RemoveContainer" containerID="6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d" Nov 01 16:21:44.148849 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:21:44.148827 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d\": container with ID starting with 6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d not found: ID does not exist" containerID="6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d" Nov 01 16:21:44.148944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.148851 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d"} err="failed to get container status \"6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d\": rpc error: code = NotFound desc = could not find container \"6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d\": container with ID starting with 6a8be4a84b52936f64ca5421336bb2dd3cbdf22739b021cdcb7fcaef1d33b70d not found: ID does not exist" Nov 01 16:21:44.155394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.155369 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z"] Nov 01 16:21:44.175770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.160782 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-10b00a-openshift-default-55c579877c-krd5z"] Nov 01 16:21:44.461998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.461904 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" path="/var/lib/kubelet/pods/9c921e79-5128-4b20-bdc3-302e60b3e5d6/volumes" Nov 01 16:21:44.634285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.634251 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:44.675953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.675917 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4"] Nov 01 16:21:44.676898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.676878 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" Nov 01 16:21:44.676898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.676898 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" Nov 01 16:21:44.677010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.676910 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" Nov 01 16:21:44.677010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.676916 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" Nov 01 16:21:44.677119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.677028 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" Nov 01 16:21:44.677119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.677037 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c921e79-5128-4b20-bdc3-302e60b3e5d6" containerName="istio-proxy" Nov 01 16:21:44.681194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.681123 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:44.687198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.687179 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-5ccd7b\"" Nov 01 16:21:44.688235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.688218 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs"] Nov 01 16:21:44.692677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.692658 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:44.694064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.694027 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4"] Nov 01 16:21:44.701647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.701627 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs"] Nov 01 16:21:44.725804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.725725 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:44.802658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.800243 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1a22e-1a29-40fe-ae79-9b3f717fc392-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs\" (UID: \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:44.804705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.803941 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m76rj\" (UniqueName: \"kubernetes.io/projected/f82a8513-36f7-4799-a7c7-0c685a52631d-kube-api-access-m76rj\") pod \"payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4\" (UID: \"f82a8513-36f7-4799-a7c7-0c685a52631d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:44.804705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.804101 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a8513-36f7-4799-a7c7-0c685a52631d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4\" (UID: \"f82a8513-36f7-4799-a7c7-0c685a52631d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:44.804705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.804364 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztrj9\" (UniqueName: \"kubernetes.io/projected/fde1a22e-1a29-40fe-ae79-9b3f717fc392-kube-api-access-ztrj9\") pod \"payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs\" (UID: \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:44.911876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.906447 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1a22e-1a29-40fe-ae79-9b3f717fc392-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs\" (UID: \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:44.911876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.906541 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m76rj\" (UniqueName: \"kubernetes.io/projected/f82a8513-36f7-4799-a7c7-0c685a52631d-kube-api-access-m76rj\") pod \"payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4\" (UID: \"f82a8513-36f7-4799-a7c7-0c685a52631d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:44.911876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.906603 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a8513-36f7-4799-a7c7-0c685a52631d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4\" (UID: \"f82a8513-36f7-4799-a7c7-0c685a52631d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:44.911876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.906777 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztrj9\" (UniqueName: \"kubernetes.io/projected/fde1a22e-1a29-40fe-ae79-9b3f717fc392-kube-api-access-ztrj9\") pod \"payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs\" (UID: \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:44.911876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.908256 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1a22e-1a29-40fe-ae79-9b3f717fc392-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs\" (UID: \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:44.911876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.909221 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a8513-36f7-4799-a7c7-0c685a52631d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4\" (UID: \"f82a8513-36f7-4799-a7c7-0c685a52631d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:44.925615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.925548 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztrj9\" (UniqueName: \"kubernetes.io/projected/fde1a22e-1a29-40fe-ae79-9b3f717fc392-kube-api-access-ztrj9\") pod \"payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs\" (UID: \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:44.964710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:44.950631 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m76rj\" (UniqueName: \"kubernetes.io/projected/f82a8513-36f7-4799-a7c7-0c685a52631d-kube-api-access-m76rj\") pod \"payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4\" (UID: \"f82a8513-36f7-4799-a7c7-0c685a52631d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:45.049093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:45.049012 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:45.056383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:45.056354 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:45.142383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:45.141842 4573 generic.go:356] "Generic (PLEG): container finished" podID="1ed0c712-d79d-4c4c-b3ce-7242c785153c" containerID="0e9dcf30619159407c9bb625e0432cc8c7774a4aedb5679b5db70d6f5b8931d6" exitCode=0 Nov 01 16:21:45.142383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:45.141929 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" event={"ID":"1ed0c712-d79d-4c4c-b3ce-7242c785153c","Type":"ContainerDied","Data":"0e9dcf30619159407c9bb625e0432cc8c7774a4aedb5679b5db70d6f5b8931d6"} Nov 01 16:21:45.293867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:45.293832 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:45.347420 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:21:45.347362 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82a8513_36f7_4799_a7c7_0c685a52631d.slice/crio-ad299db0f807c969201a09a9adcba8b5a5758e8b4ff6c4b4d914f71362d9f5d6 WatchSource:0}: Error finding container ad299db0f807c969201a09a9adcba8b5a5758e8b4ff6c4b4d914f71362d9f5d6: Status 404 returned error can't find the container with id ad299db0f807c969201a09a9adcba8b5a5758e8b4ff6c4b4d914f71362d9f5d6 Nov 01 16:21:45.347599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:45.347536 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4"] Nov 01 16:21:45.353069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:45.353046 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs"] Nov 01 16:21:45.385566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:45.385535 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:46.082577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.082545 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:46.165291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.165238 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" event={"ID":"f82a8513-36f7-4799-a7c7-0c685a52631d","Type":"ContainerStarted","Data":"465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67"} Nov 01 16:21:46.165491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.165309 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" event={"ID":"f82a8513-36f7-4799-a7c7-0c685a52631d","Type":"ContainerStarted","Data":"ad299db0f807c969201a09a9adcba8b5a5758e8b4ff6c4b4d914f71362d9f5d6"} Nov 01 16:21:46.165491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.165385 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:46.171608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.171571 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" event={"ID":"fde1a22e-1a29-40fe-ae79-9b3f717fc392","Type":"ContainerStarted","Data":"5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e"} Nov 01 16:21:46.171898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.171865 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" event={"ID":"fde1a22e-1a29-40fe-ae79-9b3f717fc392","Type":"ContainerStarted","Data":"f5e9ebe1421bcd233bbb53ec3b9dfeff1a38d189f50d02710df5f1e1715dc7aa"} Nov 01 16:21:46.189933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.188880 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" podStartSLOduration=2.188866241 podStartE2EDuration="2.188866241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:46.181483327 +0000 UTC m=+2043.884649266" watchObservedRunningTime="2026-11-01 16:21:46.188866241 +0000 UTC m=+2043.892032177" Nov 01 16:21:46.200330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.200260 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" podStartSLOduration=2.200247439 podStartE2EDuration="2.200247439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:46.198484867 +0000 UTC m=+2043.901650814" watchObservedRunningTime="2026-11-01 16:21:46.200247439 +0000 UTC m=+2043.903413374" Nov 01 16:21:46.408927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.408900 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:46.473779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.473709 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-xks-ca\") pod \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " Nov 01 16:21:46.473943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.473811 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-openshift-ca\") pod \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " Nov 01 16:21:46.473943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.473890 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ed0c712-d79d-4c4c-b3ce-7242c785153c-kube-api-access-kdpwb\" (UniqueName: \"kubernetes.io/projected/1ed0c712-d79d-4c4c-b3ce-7242c785153c-kube-api-access-kdpwb\") pod \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\" (UID: \"1ed0c712-d79d-4c4c-b3ce-7242c785153c\") " Nov 01 16:21:46.474430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.474402 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-openshift-ca" pod "1ed0c712-d79d-4c4c-b3ce-7242c785153c" (UID: "1ed0c712-d79d-4c4c-b3ce-7242c785153c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:21:46.474612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.474595 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:46.476022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.475995 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-xks-ca" pod "1ed0c712-d79d-4c4c-b3ce-7242c785153c" (UID: "1ed0c712-d79d-4c4c-b3ce-7242c785153c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:21:46.477193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.477164 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ed0c712-d79d-4c4c-b3ce-7242c785153c-kube-api-access-kdpwb" pod "1ed0c712-d79d-4c4c-b3ce-7242c785153c" (UID: "1ed0c712-d79d-4c4c-b3ce-7242c785153c"). InnerVolumeSpecName "kube-api-access-kdpwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:21:46.576445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.576412 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ed0c712-d79d-4c4c-b3ce-7242c785153c-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:46.576445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.576441 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdpwb\" (UniqueName: \"kubernetes.io/projected/1ed0c712-d79d-4c4c-b3ce-7242c785153c-kube-api-access-kdpwb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:21:46.773364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:46.773226 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:47.181048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.181016 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" Nov 01 16:21:47.181461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.181065 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l" event={"ID":"1ed0c712-d79d-4c4c-b3ce-7242c785153c","Type":"ContainerDied","Data":"1582631eb5316924ed73b1095d38e4b3d55b829869f2964a2bb6538fa2f5b478"} Nov 01 16:21:47.181461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.181117 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1582631eb5316924ed73b1095d38e4b3d55b829869f2964a2bb6538fa2f5b478" Nov 01 16:21:47.181586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.181564 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:47.466171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.466085 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7"] Nov 01 16:21:47.471586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.467232 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1ed0c712-d79d-4c4c-b3ce-7242c785153c" containerName="revoke-keys" Nov 01 16:21:47.471586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.467258 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ed0c712-d79d-4c4c-b3ce-7242c785153c" containerName="revoke-keys" Nov 01 16:21:47.471586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.467459 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ed0c712-d79d-4c4c-b3ce-7242c785153c" containerName="revoke-keys" Nov 01 16:21:47.471586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.467930 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:47.471948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.471811 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.489116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.489087 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7"] Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.490505 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.490648 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.490783 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.490927 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.490976 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.491027 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n5m4\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-kube-api-access-8n5m4\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.491078 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.491104 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.494978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.491126 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d13c4b2-be53-42da-8450-88ccb0523bf1-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.562519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.562479 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c78897-openshift-default-dockercfg-q2vpn\"" Nov 01 16:21:47.592971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.592935 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.592971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.592976 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.592994 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d13c4b2-be53-42da-8450-88ccb0523bf1-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593096 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593216 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593350 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593411 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593456 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593770 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d13c4b2-be53-42da-8450-88ccb0523bf1-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593809 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593869 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.593996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.593970 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n5m4\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-kube-api-access-8n5m4\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.594093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.594070 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.594233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.594212 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.595942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.595920 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.596108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.596089 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.601027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.600999 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.601257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.601237 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n5m4\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-kube-api-access-8n5m4\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:47.852245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:47.852155 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:48.137647 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:21:48.137613 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d13c4b2_be53_42da_8450_88ccb0523bf1.slice/crio-d5604ec4d0e4d4d40783f93f9c911559151c598d516b4770958b6c1e22ae11ec WatchSource:0}: Error finding container d5604ec4d0e4d4d40783f93f9c911559151c598d516b4770958b6c1e22ae11ec: Status 404 returned error can't find the container with id d5604ec4d0e4d4d40783f93f9c911559151c598d516b4770958b6c1e22ae11ec Nov 01 16:21:48.138201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:48.138174 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7"] Nov 01 16:21:48.141071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:48.141043 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.141147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:48.141107 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.141147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:48.141137 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:21:48.190307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:48.190283 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" event={"ID":"2d13c4b2-be53-42da-8450-88ccb0523bf1","Type":"ContainerStarted","Data":"d5604ec4d0e4d4d40783f93f9c911559151c598d516b4770958b6c1e22ae11ec"} Nov 01 16:21:48.259075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:48.259039 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:49.054201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:49.054164 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:49.212785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:49.212168 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" event={"ID":"2d13c4b2-be53-42da-8450-88ccb0523bf1","Type":"ContainerStarted","Data":"213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d"} Nov 01 16:21:49.237640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:49.237574 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" podStartSLOduration=2.237556406 podStartE2EDuration="2.237556406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:21:49.233409788 +0000 UTC m=+2046.936575724" watchObservedRunningTime="2026-11-01 16:21:49.237556406 +0000 UTC m=+2046.940722343" Nov 01 16:21:49.853009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:49.852972 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:49.854652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:49.854624 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:21:49.854795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:49.854677 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 16:21:49.906536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:49.906490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:50.852999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:50.852963 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:21:50.853460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:50.853025 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 16:21:50.888168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:50.888124 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:51.602317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:51.602286 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:51.853562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:51.853484 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:21:51.853562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:51.853540 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 16:21:52.395782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:52.395717 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:52.853051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:52.852965 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:21:52.853051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:52.853024 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 16:21:53.097772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:53.097558 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:53.201590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:53.201547 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:53.298562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:53.298526 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:53.853382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:53.853346 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" start-of-body= Nov 01 16:21:53.853567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:53.853405 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.226:15021/healthz/ready\": dial tcp 10.133.0.226:15021: connect: connection refused" Nov 01 16:21:53.991376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:53.991341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:54.389859 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:21:54.389820 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:21:54.766280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:54.766199 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:54.863321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:54.863286 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:54.863819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:54.863784 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:54.864701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:54.864676 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:21:55.598412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:55.598380 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:56.276115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:56.276048 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:57.006780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:57.006697 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:57.185102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:57.185072 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:21:57.715947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:57.715895 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:58.194413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:58.194385 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:21:58.396555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:58.396506 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:58.792299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:58.792237 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:59.450796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:59.450737 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:21:59.715358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:59.715281 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k"] Nov 01 16:21:59.715521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:21:59.715470 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" podUID="e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" containerName="discovery" containerID="cri-o://295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921" gracePeriod=30 Nov 01 16:22:00.032910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.032885 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:22:00.095340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.095299 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-cacerts\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-cacerts\") pod \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " Nov 01 16:22:00.095537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.095427 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-kubeconfig\") pod \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " Nov 01 16:22:00.095537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.095452 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-ca-configmap\") pod \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " Nov 01 16:22:00.095672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.095562 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-dns-cert\") pod \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " Nov 01 16:22:00.095672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.095612 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-token\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-token\") pod \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " Nov 01 16:22:00.095672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.095642 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-kube-api-access-66pbz\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-kube-api-access-66pbz\") pod \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " Nov 01 16:22:00.095975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.095917 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-ca-configmap" pod "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" (UID: "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:00.096126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.096087 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-local-certs\") pod \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\" (UID: \"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194\") " Nov 01 16:22:00.097018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.096983 4573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-ca-configmap\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:00.098317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.098288 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-token" pod "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" (UID: "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:00.099028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.098996 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-kubeconfig" pod "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" (UID: "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:22:00.099645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.099611 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-kube-api-access-66pbz" pod "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" (UID: "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194"). InnerVolumeSpecName "kube-api-access-66pbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:00.099745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.099683 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-dns-cert" pod "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" (UID: "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:22:00.099836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.099812 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-cacerts" pod "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" (UID: "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:22:00.100108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.100083 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-local-certs" pod "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" (UID: "e0d266dd-6807-4e9c-bb5e-59fb4d3c2194"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:00.185649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.185606 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:00.198740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.198711 4573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-csr-dns-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:00.198740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.198741 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:00.198972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.198795 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66pbz\" (UniqueName: \"kubernetes.io/projected/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-kube-api-access-66pbz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:00.198972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.198811 4573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-local-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:00.198972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.198827 4573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-cacerts\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:00.198972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.198841 4573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194-istio-kubeconfig\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:00.309869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.309744 4573 generic.go:356] "Generic (PLEG): container finished" podID="e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" containerID="295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921" exitCode=0 Nov 01 16:22:00.309869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.309834 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" Nov 01 16:22:00.310099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.309783 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" event={"ID":"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194","Type":"ContainerDied","Data":"295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921"} Nov 01 16:22:00.310099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.309936 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k" event={"ID":"e0d266dd-6807-4e9c-bb5e-59fb4d3c2194","Type":"ContainerDied","Data":"fb2f232fdd32ea83fe2702ba738dabe89be41d2a8a3e6705bffd07dad08a2634"} Nov 01 16:22:00.310099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.309972 4573 scope.go:122] "RemoveContainer" containerID="295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921" Nov 01 16:22:00.332781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.332721 4573 scope.go:122] "RemoveContainer" containerID="295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921" Nov 01 16:22:00.339242 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:22:00.339087 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921\": container with ID starting with 295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921 not found: ID does not exist" containerID="295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921" Nov 01 16:22:00.339628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.339143 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921"} err="failed to get container status \"295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921\": rpc error: code = NotFound desc = could not find container \"295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921\": container with ID starting with 295e626df6534e93dd23e4cf18b9ca635786fcdb7ffee5aa1ecf53d8a9323921 not found: ID does not exist" Nov 01 16:22:00.340954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.340918 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k"] Nov 01 16:22:00.345492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.345472 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rtv2k"] Nov 01 16:22:00.459113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.459083 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" path="/var/lib/kubelet/pods/e0d266dd-6807-4e9c-bb5e-59fb4d3c2194/volumes" Nov 01 16:22:00.923006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.922971 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq"] Nov 01 16:22:00.924102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.924076 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" containerName="discovery" Nov 01 16:22:00.924102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.924102 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" containerName="discovery" Nov 01 16:22:00.924358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.924341 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0d266dd-6807-4e9c-bb5e-59fb4d3c2194" containerName="discovery" Nov 01 16:22:00.931663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.931631 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:00.939352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.939324 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq"] Nov 01 16:22:00.939943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:00.939918 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-5ccd7b-serving-cert\"" Nov 01 16:22:01.005180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.005113 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:01.010850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.010815 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.011082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.010911 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn4kc\" (UniqueName: \"kubernetes.io/projected/ec6665a5-bce9-41b5-98ca-d7a46e344e62-kube-api-access-wn4kc\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.011082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.010954 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6665a5-bce9-41b5-98ca-d7a46e344e62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.011082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.010981 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.085825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.085793 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj"] Nov 01 16:22:01.086186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.086123 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" containerID="cri-o://04aad2811242bc854b88a1bc6824d45cf8b2e4acf142a27b13892cd67d913ed0" gracePeriod=30 Nov 01 16:22:01.113048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.111341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:01.113048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.112468 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn4kc\" (UniqueName: \"kubernetes.io/projected/ec6665a5-bce9-41b5-98ca-d7a46e344e62-kube-api-access-wn4kc\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.113048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.112516 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6665a5-bce9-41b5-98ca-d7a46e344e62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.113048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.112551 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.113048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.112904 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.113976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.113940 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6665a5-bce9-41b5-98ca-d7a46e344e62-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.115922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.115900 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.116032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.115922 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.124966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.124916 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn4kc\" (UniqueName: \"kubernetes.io/projected/ec6665a5-bce9-41b5-98ca-d7a46e344e62-kube-api-access-wn4kc\") pod \"maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.207482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.207407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:01.269499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.269471 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:01.685257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.685204 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq"] Nov 01 16:22:01.690833 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:22:01.690800 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6665a5_bce9_41b5_98ca_d7a46e344e62.slice/crio-1c1028fc4944e5b0bfca74f96db43186d6ece3b008abae2d31ae87028281456c WatchSource:0}: Error finding container 1c1028fc4944e5b0bfca74f96db43186d6ece3b008abae2d31ae87028281456c: Status 404 returned error can't find the container with id 1c1028fc4944e5b0bfca74f96db43186d6ece3b008abae2d31ae87028281456c Nov 01 16:22:01.985506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:01.985422 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:02.373478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:02.373442 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" event={"ID":"ec6665a5-bce9-41b5-98ca-d7a46e344e62","Type":"ContainerStarted","Data":"1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d"} Nov 01 16:22:02.373647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:02.373503 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" event={"ID":"ec6665a5-bce9-41b5-98ca-d7a46e344e62","Type":"ContainerStarted","Data":"1c1028fc4944e5b0bfca74f96db43186d6ece3b008abae2d31ae87028281456c"} Nov 01 16:22:02.373647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:02.373557 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:02.393162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:02.393115 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" podStartSLOduration=1.969451735 podStartE2EDuration="2.393101038s" totalImagesPullingTime="423.649303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:02.390531576 +0000 UTC m=+2060.093697515" watchObservedRunningTime="2026-11-01 16:22:02.393101038 +0000 UTC m=+2060.096266973" Nov 01 16:22:02.776403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:02.776319 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:03.587233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:03.587194 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:04.288440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:04.288407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:04.982366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:04.982325 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:05.639673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:05.639633 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:06.283279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.283239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:06.410640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.410583 4573 generic.go:356] "Generic (PLEG): container finished" podID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerID="04aad2811242bc854b88a1bc6824d45cf8b2e4acf142a27b13892cd67d913ed0" exitCode=0 Nov 01 16:22:06.410846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.410802 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" event={"ID":"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2","Type":"ContainerDied","Data":"04aad2811242bc854b88a1bc6824d45cf8b2e4acf142a27b13892cd67d913ed0"} Nov 01 16:22:06.410908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.410877 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" event={"ID":"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2","Type":"ContainerDied","Data":"610ae9ca557ed456563810213fc88e2665675810fd6b67648b991eee3f54d137"} Nov 01 16:22:06.410952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.410921 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="610ae9ca557ed456563810213fc88e2665675810fd6b67648b991eee3f54d137" Nov 01 16:22:06.441648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.441619 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:22:06.548538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.548504 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-podinfo\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.548775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.548546 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-socket\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.548775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.548618 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-kube-api-access-btqfk\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.548775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.548719 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-credential-socket\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.548998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.548878 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-token\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-token\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.549068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.549003 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-certs\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.549068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.549049 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-data\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.549185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.549084 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istiod-ca-cert\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.549240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.549190 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-envoy\") pod \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\" (UID: \"bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2\") " Nov 01 16:22:06.549296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.549232 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-socket" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.549583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.549551 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-certs" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.549896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.549828 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.549896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.549852 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.550259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.550223 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istiod-ca-cert" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:06.550826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.550800 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-data" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.550967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.550904 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-credential-socket" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.552009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.551986 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-podinfo" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:22:06.552975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.552943 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-token" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:06.553094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.552969 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-envoy" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:06.553735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.553712 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-kube-api-access-btqfk" pod "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" (UID: "bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2"). InnerVolumeSpecName "kube-api-access-btqfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:06.651405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.651322 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.651405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.651355 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.651405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.651364 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.651405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.651373 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.651405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.651383 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.651405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.651392 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btqfk\" (UniqueName: \"kubernetes.io/projected/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-kube-api-access-btqfk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.651405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.651401 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:06.989510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:06.989426 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:07.418191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:07.418160 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj" Nov 01 16:22:07.444060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:07.444029 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj"] Nov 01 16:22:07.449053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:07.449030 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-rmmkj"] Nov 01 16:22:07.665855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:07.665818 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:08.354283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:08.354254 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:08.386262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:08.386231 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:22:08.481171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:08.481137 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" path="/var/lib/kubelet/pods/bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2/volumes" Nov 01 16:22:08.982938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:08.982901 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:09.571000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:09.570961 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:09.758772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:09.757485 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:09.882170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:09.879440 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:09.913075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:09.913043 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l"] Nov 01 16:22:09.928797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:09.921047 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a4240a-tcd4l"] Nov 01 16:22:10.137371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:10.137283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:10.371349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:10.371318 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:10.459331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:10.459252 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ed0c712-d79d-4c4c-b3ce-7242c785153c" path="/var/lib/kubelet/pods/1ed0c712-d79d-4c4c-b3ce-7242c785153c/volumes" Nov 01 16:22:10.867564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:10.867539 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:11.590525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:11.590495 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:12.351326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:12.351293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:13.049408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:13.049350 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:13.679845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:13.679814 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:14.377421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:14.377381 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:15.065989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:15.065952 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:15.728458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:15.728429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:16.463566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:16.463529 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:16.574800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:16.574737 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:16.670645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:16.670612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:16.870891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:16.870853 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8"] Nov 01 16:22:16.871364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:16.871125 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" containerID="cri-o://951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9" gracePeriod=30 Nov 01 16:22:16.905611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:16.905556 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:17.282318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.282220 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:17.380430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.380398 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml"] Nov 01 16:22:17.390007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.389973 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" Nov 01 16:22:17.390007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.390004 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" Nov 01 16:22:17.390267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.390248 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc95c5b3-3ef7-4e8a-9082-8cc0b54a27d2" containerName="istio-proxy" Nov 01 16:22:17.397016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.396987 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml"] Nov 01 16:22:17.397201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.397100 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.427160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.426329 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:22:17.427160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.426474 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:22:17.442349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.442305 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.442501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.442378 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.442636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.442621 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bkph\" (UniqueName: \"kubernetes.io/projected/475b51b4-57a5-4507-8ec0-9edb709329a4-kube-api-access-9bkph\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.544516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.544440 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bkph\" (UniqueName: \"kubernetes.io/projected/475b51b4-57a5-4507-8ec0-9edb709329a4-kube-api-access-9bkph\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.544701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.544545 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.544701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.544586 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.545289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.545263 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.546971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.546948 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.552195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.552168 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bkph\" (UniqueName: \"kubernetes.io/projected/475b51b4-57a5-4507-8ec0-9edb709329a4-kube-api-access-9bkph\") pod \"maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:17.735577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:17.735549 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:18.022498 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:22:18.022466 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475b51b4_57a5_4507_8ec0_9edb709329a4.slice/crio-85e687d78909a5f24059b05c9f23db67df9f13079ebba764b3be81744cd60f79 WatchSource:0}: Error finding container 85e687d78909a5f24059b05c9f23db67df9f13079ebba764b3be81744cd60f79: Status 404 returned error can't find the container with id 85e687d78909a5f24059b05c9f23db67df9f13079ebba764b3be81744cd60f79 Nov 01 16:22:18.023499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:18.023468 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml"] Nov 01 16:22:18.265821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:18.265784 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:18.577627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:18.577591 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" event={"ID":"475b51b4-57a5-4507-8ec0-9edb709329a4","Type":"ContainerStarted","Data":"7be0fbcd8582656a8d92d7fff31ad32d95a480b0dd7c4091e163cae414514b39"} Nov 01 16:22:18.577813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:18.577651 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" event={"ID":"475b51b4-57a5-4507-8ec0-9edb709329a4","Type":"ContainerStarted","Data":"85e687d78909a5f24059b05c9f23db67df9f13079ebba764b3be81744cd60f79"} Nov 01 16:22:18.594100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:18.594049 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" podStartSLOduration=1.5940347620000002 podStartE2EDuration="1.594034762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:18.591861338 +0000 UTC m=+2076.295027275" watchObservedRunningTime="2026-11-01 16:22:18.594034762 +0000 UTC m=+2076.297200695" Nov 01 16:22:18.771693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:18.771610 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:19.151448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:19.151414 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:19.254832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:19.254797 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:19.587930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:19.587898 4573 generic.go:356] "Generic (PLEG): container finished" podID="475b51b4-57a5-4507-8ec0-9edb709329a4" containerID="7be0fbcd8582656a8d92d7fff31ad32d95a480b0dd7c4091e163cae414514b39" exitCode=0 Nov 01 16:22:19.588159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:19.587979 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" event={"ID":"475b51b4-57a5-4507-8ec0-9edb709329a4","Type":"ContainerDied","Data":"7be0fbcd8582656a8d92d7fff31ad32d95a480b0dd7c4091e163cae414514b39"} Nov 01 16:22:19.689614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:19.689578 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:19.870467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:19.870432 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:20.155260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.155175 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:20.749952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.749926 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:20.854187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.854093 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:20.864854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.864823 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-xks-ca\") pod \"475b51b4-57a5-4507-8ec0-9edb709329a4\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " Nov 01 16:22:20.865135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.865118 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-openshift-ca\") pod \"475b51b4-57a5-4507-8ec0-9edb709329a4\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " Nov 01 16:22:20.865233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.865213 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475b51b4-57a5-4507-8ec0-9edb709329a4-kube-api-access-9bkph\" (UniqueName: \"kubernetes.io/projected/475b51b4-57a5-4507-8ec0-9edb709329a4-kube-api-access-9bkph\") pod \"475b51b4-57a5-4507-8ec0-9edb709329a4\" (UID: \"475b51b4-57a5-4507-8ec0-9edb709329a4\") " Nov 01 16:22:20.865485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.865458 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-openshift-ca" pod "475b51b4-57a5-4507-8ec0-9edb709329a4" (UID: "475b51b4-57a5-4507-8ec0-9edb709329a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:20.865929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.865911 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:20.867109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.867087 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-xks-ca" pod "475b51b4-57a5-4507-8ec0-9edb709329a4" (UID: "475b51b4-57a5-4507-8ec0-9edb709329a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:22:20.867479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.867253 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475b51b4-57a5-4507-8ec0-9edb709329a4-kube-api-access-9bkph" pod "475b51b4-57a5-4507-8ec0-9edb709329a4" (UID: "475b51b4-57a5-4507-8ec0-9edb709329a4"). InnerVolumeSpecName "kube-api-access-9bkph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:20.967107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.967073 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bkph\" (UniqueName: \"kubernetes.io/projected/475b51b4-57a5-4507-8ec0-9edb709329a4-kube-api-access-9bkph\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:20.967107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:20.967104 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/475b51b4-57a5-4507-8ec0-9edb709329a4-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:21.605675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:21.605638 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" Nov 01 16:22:21.605675 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:21.605661 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml" event={"ID":"475b51b4-57a5-4507-8ec0-9edb709329a4","Type":"ContainerDied","Data":"85e687d78909a5f24059b05c9f23db67df9f13079ebba764b3be81744cd60f79"} Nov 01 16:22:21.606170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:21.605710 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e687d78909a5f24059b05c9f23db67df9f13079ebba764b3be81744cd60f79" Nov 01 16:22:21.642974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:21.642945 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:22.158093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.158062 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:22:22.284870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.284838 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-kube-api-access-7rmrc\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-kube-api-access-7rmrc\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.284949 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-data\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285001 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/572793d5-f154-417a-850e-12e176c14413-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572793d5-f154-417a-850e-12e176c14413-istiod-ca-cert\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285048 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/572793d5-f154-417a-850e-12e176c14413-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572793d5-f154-417a-850e-12e176c14413-istio-podinfo\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285113 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-envoy\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285144 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-istio-token\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-istio-token\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285208 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-socket\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285260 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-certs\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285340 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-credential-socket\") pod \"572793d5-f154-417a-850e-12e176c14413\" (UID: \"572793d5-f154-417a-850e-12e176c14413\") " Nov 01 16:22:22.285933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285255 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-data" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.285933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285454 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572793d5-f154-417a-850e-12e176c14413-istiod-ca-cert" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:22.285933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285556 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-socket" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.285933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285713 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-certs" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.285933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.285903 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-credential-socket" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.286491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.286102 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.286491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.286125 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.286491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.286138 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.286491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.286152 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.286491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.286165 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/572793d5-f154-417a-850e-12e176c14413-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.287714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.287689 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-kube-api-access-7rmrc" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "kube-api-access-7rmrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:22.287889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.287722 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-istio-token" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:22.287889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.287862 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/572793d5-f154-417a-850e-12e176c14413-istio-podinfo" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:22:22.288120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.288093 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-envoy" pod "572793d5-f154-417a-850e-12e176c14413" (UID: "572793d5-f154-417a-850e-12e176c14413"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:22.365730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.365689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:22.387705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.387624 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/572793d5-f154-417a-850e-12e176c14413-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.387705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.387653 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.387705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.387683 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rmrc\" (UniqueName: \"kubernetes.io/projected/572793d5-f154-417a-850e-12e176c14413-kube-api-access-7rmrc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.387705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.387694 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/572793d5-f154-417a-850e-12e176c14413-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:22.616230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.616083 4573 generic.go:356] "Generic (PLEG): container finished" podID="572793d5-f154-417a-850e-12e176c14413" containerID="951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9" exitCode=0 Nov 01 16:22:22.616230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.616165 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" Nov 01 16:22:22.616800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.616339 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" event={"ID":"572793d5-f154-417a-850e-12e176c14413","Type":"ContainerDied","Data":"951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9"} Nov 01 16:22:22.616800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.616396 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8" event={"ID":"572793d5-f154-417a-850e-12e176c14413","Type":"ContainerDied","Data":"800f97bd193d1badc1a60ba43f92fcf07f171a6ae162a9f6efe702e4ad195e4d"} Nov 01 16:22:22.616800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.616429 4573 scope.go:122] "RemoveContainer" containerID="951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9" Nov 01 16:22:22.628531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.628497 4573 scope.go:122] "RemoveContainer" containerID="951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9" Nov 01 16:22:22.628858 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:22:22.628835 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9\": container with ID starting with 951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9 not found: ID does not exist" containerID="951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9" Nov 01 16:22:22.628929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.628866 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9"} err="failed to get container status \"951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9\": rpc error: code = NotFound desc = could not find container \"951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9\": container with ID starting with 951ba6fe64c60117a70baf939cd5991a2e2670347aa9fd9f90f4cfcd1aae87a9 not found: ID does not exist" Nov 01 16:22:22.636050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.636028 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8"] Nov 01 16:22:22.640396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:22.640346 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a4240a-openshift-default-8449487f59-lmth8"] Nov 01 16:22:23.248927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:23.248884 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:23.617209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:23.617167 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:23.716550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:23.716517 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:24.138025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.137989 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:24.364903 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:22:24.364867 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:22:24.459160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.459088 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="572793d5-f154-417a-850e-12e176c14413" path="/var/lib/kubelet/pods/572793d5-f154-417a-850e-12e176c14413/volumes" Nov 01 16:22:24.806110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.806012 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:24.881718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.881685 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6"] Nov 01 16:22:24.882420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.882405 4573 cpu_manager.go:414] "containerMap: removing container" podUID="475b51b4-57a5-4507-8ec0-9edb709329a4" containerName="revoke-keys" Nov 01 16:22:24.882474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.882421 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475b51b4-57a5-4507-8ec0-9edb709329a4" containerName="revoke-keys" Nov 01 16:22:24.882474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.882461 4573 cpu_manager.go:414] "containerMap: removing container" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" Nov 01 16:22:24.882474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.882466 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" Nov 01 16:22:24.882581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.882567 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="475b51b4-57a5-4507-8ec0-9edb709329a4" containerName="revoke-keys" Nov 01 16:22:24.882616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.882580 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="572793d5-f154-417a-850e-12e176c14413" containerName="istio-proxy" Nov 01 16:22:24.888940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.888917 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:24.892799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.892262 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-c8537c\"" Nov 01 16:22:24.892799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.892332 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-c8537c-dockercfg-xj89r\"" Nov 01 16:22:24.893527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.893505 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6"] Nov 01 16:22:24.908401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.908370 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:24.931861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.931823 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9"] Nov 01 16:22:24.932058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.932021 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba7eb6-54bb-45e1-9522-886df60e939f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6\" (UID: \"5eba7eb6-54bb-45e1-9522-886df60e939f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:24.932238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.932212 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85fsw\" (UniqueName: \"kubernetes.io/projected/5eba7eb6-54bb-45e1-9522-886df60e939f-kube-api-access-85fsw\") pod \"payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6\" (UID: \"5eba7eb6-54bb-45e1-9522-886df60e939f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:24.936588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.936563 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:24.944631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:24.944601 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9"] Nov 01 16:22:25.063601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.063508 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba7eb6-54bb-45e1-9522-886df60e939f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6\" (UID: \"5eba7eb6-54bb-45e1-9522-886df60e939f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:25.063863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.063689 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwk22\" (UniqueName: \"kubernetes.io/projected/75996380-b13c-4d2a-8a70-494c0bf2ac06-kube-api-access-rwk22\") pod \"payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9\" (UID: \"75996380-b13c-4d2a-8a70-494c0bf2ac06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:25.063863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.063734 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85fsw\" (UniqueName: \"kubernetes.io/projected/5eba7eb6-54bb-45e1-9522-886df60e939f-kube-api-access-85fsw\") pod \"payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6\" (UID: \"5eba7eb6-54bb-45e1-9522-886df60e939f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:25.066772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.064683 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba7eb6-54bb-45e1-9522-886df60e939f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6\" (UID: \"5eba7eb6-54bb-45e1-9522-886df60e939f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:25.066772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.065259 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75996380-b13c-4d2a-8a70-494c0bf2ac06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9\" (UID: \"75996380-b13c-4d2a-8a70-494c0bf2ac06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:25.126594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.125528 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85fsw\" (UniqueName: \"kubernetes.io/projected/5eba7eb6-54bb-45e1-9522-886df60e939f-kube-api-access-85fsw\") pod \"payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6\" (UID: \"5eba7eb6-54bb-45e1-9522-886df60e939f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:25.128038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.127630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:25.166959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.166923 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwk22\" (UniqueName: \"kubernetes.io/projected/75996380-b13c-4d2a-8a70-494c0bf2ac06-kube-api-access-rwk22\") pod \"payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9\" (UID: \"75996380-b13c-4d2a-8a70-494c0bf2ac06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:25.167187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.167057 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75996380-b13c-4d2a-8a70-494c0bf2ac06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9\" (UID: \"75996380-b13c-4d2a-8a70-494c0bf2ac06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:25.167851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.167828 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75996380-b13c-4d2a-8a70-494c0bf2ac06-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9\" (UID: \"75996380-b13c-4d2a-8a70-494c0bf2ac06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:25.199714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.199340 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwk22\" (UniqueName: \"kubernetes.io/projected/75996380-b13c-4d2a-8a70-494c0bf2ac06-kube-api-access-rwk22\") pod \"payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9\" (UID: \"75996380-b13c-4d2a-8a70-494c0bf2ac06\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:25.209335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.209310 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:25.258208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.258178 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:25.506444 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:22:25.506411 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eba7eb6_54bb_45e1_9522_886df60e939f.slice/crio-c1d182ee5623209cb018bd3caf0948fb1db54852e7abef4f2fb0d4b2139c16e1 WatchSource:0}: Error finding container c1d182ee5623209cb018bd3caf0948fb1db54852e7abef4f2fb0d4b2139c16e1: Status 404 returned error can't find the container with id c1d182ee5623209cb018bd3caf0948fb1db54852e7abef4f2fb0d4b2139c16e1 Nov 01 16:22:25.507863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.507835 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6"] Nov 01 16:22:25.589375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.589131 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9"] Nov 01 16:22:25.589375 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:22:25.589290 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75996380_b13c_4d2a_8a70_494c0bf2ac06.slice/crio-24086ef591f0b1bc345a46514241c2e0b1aed58e615a6e9c67cafff7be7db8f3 WatchSource:0}: Error finding container 24086ef591f0b1bc345a46514241c2e0b1aed58e615a6e9c67cafff7be7db8f3: Status 404 returned error can't find the container with id 24086ef591f0b1bc345a46514241c2e0b1aed58e615a6e9c67cafff7be7db8f3 Nov 01 16:22:25.646324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.646295 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" event={"ID":"75996380-b13c-4d2a-8a70-494c0bf2ac06","Type":"ContainerStarted","Data":"24086ef591f0b1bc345a46514241c2e0b1aed58e615a6e9c67cafff7be7db8f3"} Nov 01 16:22:25.647841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.647818 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" event={"ID":"5eba7eb6-54bb-45e1-9522-886df60e939f","Type":"ContainerStarted","Data":"0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373"} Nov 01 16:22:25.647953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.647870 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" event={"ID":"5eba7eb6-54bb-45e1-9522-886df60e939f","Type":"ContainerStarted","Data":"c1d182ee5623209cb018bd3caf0948fb1db54852e7abef4f2fb0d4b2139c16e1"} Nov 01 16:22:25.647953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.647917 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:25.684347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:25.684299 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" podStartSLOduration=1.684285429 podStartE2EDuration="1.684285429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:25.679083006 +0000 UTC m=+2083.382248941" watchObservedRunningTime="2026-11-01 16:22:25.684285429 +0000 UTC m=+2083.387451364" Nov 01 16:22:26.077417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:26.077386 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:26.657798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:26.657760 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" event={"ID":"75996380-b13c-4d2a-8a70-494c0bf2ac06","Type":"ContainerStarted","Data":"0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170"} Nov 01 16:22:26.658014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:26.657820 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:26.678284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:26.678235 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" podStartSLOduration=2.678223703 podStartE2EDuration="2.678223703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:26.676844423 +0000 UTC m=+2084.380010373" watchObservedRunningTime="2026-11-01 16:22:26.678223703 +0000 UTC m=+2084.381389638" Nov 01 16:22:26.930547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:26.930467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:27.556349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:27.556311 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:27.660341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:27.660307 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:27.806076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:27.806039 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:28.454572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:28.454540 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:29.179620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:29.179584 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:29.867550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:29.867509 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:30.662809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:30.662772 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:31.355696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:31.355659 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:31.886139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:31.886103 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:31.992705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:31.992668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:32.867674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:32.867629 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:33.510424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:33.510374 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:33.657220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:33.657182 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:33.780836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:33.778017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:33.914602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:33.914568 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml"] Nov 01 16:22:33.919413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:33.919376 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0c091d-vkwml"] Nov 01 16:22:34.008531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:34.008494 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:34.351257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:34.351221 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:34.459810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:34.459779 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475b51b4-57a5-4507-8ec0-9edb709329a4" path="/var/lib/kubelet/pods/475b51b4-57a5-4507-8ec0-9edb709329a4/volumes" Nov 01 16:22:34.736811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:34.736778 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:22:34.886870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:34.886828 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:35.830467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:35.830436 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:36.661318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:36.661291 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:22:36.881800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:36.881765 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:37.669455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:37.669427 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:22:38.233398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:38.233362 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:39.324873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:39.324839 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:40.556596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:40.556562 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:40.655391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:40.655357 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:40.907666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:40.907633 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm"] Nov 01 16:22:40.907960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:40.907924 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" containerID="cri-o://454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10" gracePeriod=30 Nov 01 16:22:40.922033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:40.922004 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:41.539613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:41.539583 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:42.364768 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.364716 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd"] Nov 01 16:22:42.373925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.373895 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.375667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.375479 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd"] Nov 01 16:22:42.378010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.377984 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-c8537c-serving-cert\"" Nov 01 16:22:42.380114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.380088 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:42.488730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.488680 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:42.497112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.497080 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6dz\" (UniqueName: \"kubernetes.io/projected/853b6f9e-33a9-448c-b988-155a08666a2f-kube-api-access-dv6dz\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.497570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.497536 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.497708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.497633 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853b6f9e-33a9-448c-b988-155a08666a2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.497708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.497695 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.588022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.587984 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:42.599192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.599165 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.599405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.599208 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853b6f9e-33a9-448c-b988-155a08666a2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.599405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.599241 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.599405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.599280 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6dz\" (UniqueName: \"kubernetes.io/projected/853b6f9e-33a9-448c-b988-155a08666a2f-kube-api-access-dv6dz\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.599941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.599910 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853b6f9e-33a9-448c-b988-155a08666a2f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.601920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.601896 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-c8537c-serving-cert\"" Nov 01 16:22:42.602061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.602011 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.608087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.608066 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6dz\" (UniqueName: \"kubernetes.io/projected/853b6f9e-33a9-448c-b988-155a08666a2f-kube-api-access-dv6dz\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.611869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.611842 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:42.694827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.694740 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:43.002832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:42.987168 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd"] Nov 01 16:22:43.236605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.236528 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:22:43.252286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.252264 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:22:43.254065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.254041 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:22:43.264436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.264408 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:22:43.377238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.377203 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:43.653590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.653554 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69"] Nov 01 16:22:43.663151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.663125 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.669803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.669778 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69"] Nov 01 16:22:43.670657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.670608 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f47890\"/\"rate-test-model-f47890-kserve-self-signed-certs\"" Nov 01 16:22:43.670657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.670642 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-f47890\"/\"default-dockercfg-46rg7\"" Nov 01 16:22:43.670657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.670657 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f47890\"/\"kube-root-ca.crt\"" Nov 01 16:22:43.670923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.670696 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-f47890\"/\"openshift-service-ca.crt\"" Nov 01 16:22:43.699474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.699445 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:43.716138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.716108 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-home\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.716309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.716180 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d3c991-2024-4b8f-b402-9d217856f58b-tls-certs\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.716309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.716263 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-tmp-dir\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.716389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.716350 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-model-cache\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.716429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.716399 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnkcg\" (UniqueName: \"kubernetes.io/projected/24d3c991-2024-4b8f-b402-9d217856f58b-kube-api-access-xnkcg\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.716470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.716451 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-dshm\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.817525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.817421 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-home\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.817525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.817486 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d3c991-2024-4b8f-b402-9d217856f58b-tls-certs\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.817525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.817524 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-tmp-dir\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.817862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.817594 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-model-cache\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.817862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.817614 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnkcg\" (UniqueName: \"kubernetes.io/projected/24d3c991-2024-4b8f-b402-9d217856f58b-kube-api-access-xnkcg\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.817862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.817675 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-dshm\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.818045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.818009 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-model-cache\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.818210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.818181 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-tmp-dir\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.818210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.818195 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-home\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.820079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.820054 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-dshm\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.820265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.820242 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d3c991-2024-4b8f-b402-9d217856f58b-tls-certs\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.825880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.825853 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnkcg\" (UniqueName: \"kubernetes.io/projected/24d3c991-2024-4b8f-b402-9d217856f58b-kube-api-access-xnkcg\") pod \"rate-test-model-f47890-kserve-784796cc45-6gr69\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:43.920771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.920311 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" event={"ID":"853b6f9e-33a9-448c-b988-155a08666a2f","Type":"ContainerStarted","Data":"3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f"} Nov 01 16:22:43.920771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.920417 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" event={"ID":"853b6f9e-33a9-448c-b988-155a08666a2f","Type":"ContainerStarted","Data":"aa638a854deda35be0d25d14970fe40587a71f00111ff5807bd68126e04fe860"} Nov 01 16:22:43.924770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.921438 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:43.945830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.944196 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" podStartSLOduration=1.467272428 podStartE2EDuration="1.944179013s" totalImagesPullingTime="476.906585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:43.942170256 +0000 UTC m=+2101.645336194" watchObservedRunningTime="2026-11-01 16:22:43.944179013 +0000 UTC m=+2101.647344950" Nov 01 16:22:43.983097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:43.983068 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:44.295522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:44.295455 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69"] Nov 01 16:22:44.359485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:44.359454 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:44.930707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:44.930668 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" event={"ID":"24d3c991-2024-4b8f-b402-9d217856f58b","Type":"ContainerStarted","Data":"6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660"} Nov 01 16:22:44.931149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:44.930720 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" event={"ID":"24d3c991-2024-4b8f-b402-9d217856f58b","Type":"ContainerStarted","Data":"2e9d6d754c1634e430ec8c21da204f7f58862530811094f5612615aae7242151"} Nov 01 16:22:44.949884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:44.949840 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" podStartSLOduration=1.949826764 podStartE2EDuration="1.949826764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:44.946017794 +0000 UTC m=+2102.649183729" watchObservedRunningTime="2026-11-01 16:22:44.949826764 +0000 UTC m=+2102.652992714" Nov 01 16:22:44.983633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:44.983606 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:45.157429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:45.157390 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:45.794717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:45.794682 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:46.192687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.192659 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:22:46.270735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.270704 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-kube-api-access-kccrj\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-kube-api-access-kccrj\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.270922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.270746 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-socket\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.270922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.270843 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-podinfo\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.271051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.270932 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1febdd5f-7998-4501-a41d-d87ef58a3703-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febdd5f-7998-4501-a41d-d87ef58a3703-istiod-ca-cert\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.271051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.270965 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-certs\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.271200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271061 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-envoy\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.271200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271065 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-socket" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:46.271200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271090 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-data\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.271200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271164 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-token\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-token\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.271391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271294 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-credential-socket\") pod \"1febdd5f-7998-4501-a41d-d87ef58a3703\" (UID: \"1febdd5f-7998-4501-a41d-d87ef58a3703\") " Nov 01 16:22:46.271428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271400 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-data" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:46.271654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271631 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-certs" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:46.271802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271650 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-credential-socket" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:46.271802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.271665 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1febdd5f-7998-4501-a41d-d87ef58a3703-istiod-ca-cert" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:22:46.272203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.272178 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1febdd5f-7998-4501-a41d-d87ef58a3703-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.272340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.272208 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.272340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.272223 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.272340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.272237 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.272340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.272251 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.273256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.273229 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-kube-api-access-kccrj" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "kube-api-access-kccrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:46.273467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.273449 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-podinfo" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:22:46.273903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.273887 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-envoy" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:22:46.274226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.274203 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-token" pod "1febdd5f-7998-4501-a41d-d87ef58a3703" (UID: "1febdd5f-7998-4501-a41d-d87ef58a3703"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:22:46.373485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.373454 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.373485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.373481 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.373673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.373491 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.373673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.373500 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kccrj\" (UniqueName: \"kubernetes.io/projected/1febdd5f-7998-4501-a41d-d87ef58a3703-kube-api-access-kccrj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:22:46.517152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.517119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:46.787160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.787077 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c"] Nov 01 16:22:46.788013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.787987 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" Nov 01 16:22:46.788013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.788012 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" Nov 01 16:22:46.788277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.788261 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" Nov 01 16:22:46.788343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.788288 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" Nov 01 16:22:46.792637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.792620 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.801743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.801720 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-7225e5-openshift-default-dockercfg-bgkdt\"" Nov 01 16:22:46.810867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.810843 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c"] Nov 01 16:22:46.895485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.895451 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.895654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.895627 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3149b1-0513-478e-8c0f-633766afc158-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.895704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.895657 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.895854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.895832 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.895903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.895864 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.895903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.895882 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.896006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.895987 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.896059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.896043 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3149b1-0513-478e-8c0f-633766afc158-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.896118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.896103 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk6h9\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-kube-api-access-pk6h9\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.951950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.951921 4573 generic.go:356] "Generic (PLEG): container finished" podID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerID="454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10" exitCode=0 Nov 01 16:22:46.952092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.951974 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" Nov 01 16:22:46.952092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.952002 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" event={"ID":"1febdd5f-7998-4501-a41d-d87ef58a3703","Type":"ContainerDied","Data":"454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10"} Nov 01 16:22:46.952092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.952059 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm" event={"ID":"1febdd5f-7998-4501-a41d-d87ef58a3703","Type":"ContainerDied","Data":"1565520f9668caabe0d75f1cab5942a35f50940bbbf87af818c09f11946e3757"} Nov 01 16:22:46.952223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.952090 4573 scope.go:122] "RemoveContainer" containerID="454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10" Nov 01 16:22:46.967740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.967720 4573 scope.go:122] "RemoveContainer" containerID="dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046" Nov 01 16:22:46.979126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.979093 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm"] Nov 01 16:22:46.987956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.987932 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0c091d-openshift-default-6855f44f99-srhdm"] Nov 01 16:22:46.993741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.993715 4573 scope.go:122] "RemoveContainer" containerID="454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10" Nov 01 16:22:46.994064 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:22:46.994035 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10\": container with ID starting with 454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10 not found: ID does not exist" containerID="454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10" Nov 01 16:22:46.994151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.994071 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10"} err="failed to get container status \"454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10\": rpc error: code = NotFound desc = could not find container \"454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10\": container with ID starting with 454bd565b229f477f7010b392a47851417cb1c327c6a180e5e69086f4d5f8a10 not found: ID does not exist" Nov 01 16:22:46.994151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.994089 4573 scope.go:122] "RemoveContainer" containerID="dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046" Nov 01 16:22:46.994312 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:22:46.994290 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046\": container with ID starting with dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046 not found: ID does not exist" containerID="dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046" Nov 01 16:22:46.994364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.994322 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046"} err="failed to get container status \"dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046\": rpc error: code = NotFound desc = could not find container \"dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046\": container with ID starting with dbf1329fc6452e8403ef8b4c89f25bb6d624a2cf6be64d2edf67231f99478046 not found: ID does not exist" Nov 01 16:22:46.997148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997128 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk6h9\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-kube-api-access-pk6h9\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997162 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997252 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3149b1-0513-478e-8c0f-633766afc158-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997277 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997361 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997378 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997395 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997454 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997507 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3149b1-0513-478e-8c0f-633766afc158-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.997972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.997942 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-credential-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.998080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.998048 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-certs\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.998154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.998075 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-socket\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.998215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.998178 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-data\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.998530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.998491 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3149b1-0513-478e-8c0f-633766afc158-istiod-ca-cert\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.999859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.999836 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-envoy\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:46.999972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:46.999950 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3149b1-0513-478e-8c0f-633766afc158-istio-podinfo\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:47.004744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.004721 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk6h9\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-kube-api-access-pk6h9\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:47.005107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.005091 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-istio-token\") pod \"e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:47.120318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.120285 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:47.411649 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:22:47.411616 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3149b1_0513_478e_8c0f_633766afc158.slice/crio-fb3b979e38e241d5e4f14f4ae79f6666d74fecde0025a11ab92e82127901b67e WatchSource:0}: Error finding container fb3b979e38e241d5e4f14f4ae79f6666d74fecde0025a11ab92e82127901b67e: Status 404 returned error can't find the container with id fb3b979e38e241d5e4f14f4ae79f6666d74fecde0025a11ab92e82127901b67e Nov 01 16:22:47.412244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.412218 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c"] Nov 01 16:22:47.415088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.415053 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.415225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.415137 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.415225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.415183 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:22:47.488118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.488089 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:47.963820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.963785 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" event={"ID":"8a3149b1-0513-478e-8c0f-633766afc158","Type":"ContainerStarted","Data":"2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7"} Nov 01 16:22:47.964009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.963853 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" event={"ID":"8a3149b1-0513-478e-8c0f-633766afc158","Type":"ContainerStarted","Data":"fb3b979e38e241d5e4f14f4ae79f6666d74fecde0025a11ab92e82127901b67e"} Nov 01 16:22:47.983424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:47.983376 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podStartSLOduration=1.983361063 podStartE2EDuration="1.983361063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:22:47.98267592 +0000 UTC m=+2105.685841858" watchObservedRunningTime="2026-11-01 16:22:47.983361063 +0000 UTC m=+2105.686526998" Nov 01 16:22:48.103674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:48.103635 4573 scope.go:122] "RemoveContainer" containerID="04aad2811242bc854b88a1bc6824d45cf8b2e4acf142a27b13892cd67d913ed0" Nov 01 16:22:48.121193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:48.121167 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:48.123544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:48.123522 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 01 16:22:48.123654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:48.123567 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 01 16:22:48.239174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:48.239095 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:48.458702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:48.458658 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" path="/var/lib/kubelet/pods/1febdd5f-7998-4501-a41d-d87ef58a3703/volumes" Nov 01 16:22:49.087492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:49.087421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:49.120723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:49.120691 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 01 16:22:49.120915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:49.120740 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 01 16:22:50.121652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:50.121620 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 01 16:22:50.122078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:50.121675 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 01 16:22:50.135602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:50.135573 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:50.946729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:50.946702 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:22:51.021584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:51.021546 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:51.121057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:51.121016 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 01 16:22:51.121294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:51.121093 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 01 16:22:51.993550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:51.993519 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:52.121400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:52.121362 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 01 16:22:52.121582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:52.121437 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 01 16:22:53.107160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:53.107116 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:53.121827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:53.121790 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 01 16:22:53.121983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:53.121857 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 01 16:22:53.987013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:53.986928 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:54.000440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:54.000415 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:54.060393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:54.060364 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:22:54.098577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:54.098542 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:54.121505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:54.121469 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" start-of-body= Nov 01 16:22:54.121993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:54.121544 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.233:15021/healthz/ready\": dial tcp 10.133.0.233:15021: connect: connection refused" Nov 01 16:22:54.366296 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:22:54.366239 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:22:54.863702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:54.863661 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:55.126414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:55.126342 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:55.126816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:55.126779 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:55.127431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:55.127410 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:22:55.583296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:55.583213 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:56.359479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:56.359442 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:57.089006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:57.088978 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:57.848428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:57.848392 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:58.565627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:58.565592 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:59.023365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:59.023331 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb"] Nov 01 16:22:59.027362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:59.027329 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25f05218-tfwwb"] Nov 01 16:22:59.208663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:59.208630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:22:59.897002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:22:59.896968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:00.460840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:00.460806 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="126681f3-42a3-4a99-bfba-cc2763e3970f" path="/var/lib/kubelet/pods/126681f3-42a3-4a99-bfba-cc2763e3970f/volumes" Nov 01 16:23:00.593682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:00.593647 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:01.434322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:01.434289 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:02.171216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:02.171181 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:02.942535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:02.942499 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:03.553075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.553043 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f"] Nov 01 16:23:03.553894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.553877 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" Nov 01 16:23:03.553948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.553895 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1febdd5f-7998-4501-a41d-d87ef58a3703" containerName="istio-proxy" Nov 01 16:23:03.559841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.559814 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.564624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.564599 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:23:03.568184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.568137 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:23:03.568674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.568654 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f"] Nov 01 16:23:03.598820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.598720 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:03.658440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.658397 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn"] Nov 01 16:23:03.670434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.670405 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn"] Nov 01 16:23:03.670598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.670493 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.673825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.673796 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-734859\"/\"kube-root-ca.crt\"" Nov 01 16:23:03.674067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.674046 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-734859\"/\"openshift-service-ca.crt\"" Nov 01 16:23:03.674162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.674138 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-734859\"/\"rate-test-model-734859-kserve-self-signed-certs\"" Nov 01 16:23:03.674321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.674302 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-734859\"/\"default-dockercfg-kcgs2\"" Nov 01 16:23:03.682282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.682255 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vj9h\" (UniqueName: \"kubernetes.io/projected/daa04960-f591-4f15-8c63-07e571e77dbb-kube-api-access-9vj9h\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.682493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.682392 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.682493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.682470 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.752904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.752855 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:03.783512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.783477 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-model-cache\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.783716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.783677 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-dshm\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.783858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.783716 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vj9h\" (UniqueName: \"kubernetes.io/projected/daa04960-f591-4f15-8c63-07e571e77dbb-kube-api-access-9vj9h\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.783927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.783855 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-tmp-dir\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.784006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.783986 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.784085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.784071 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.784141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.784121 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-home\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.784272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.784251 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98kg\" (UniqueName: \"kubernetes.io/projected/38865718-c2c2-48c3-aa91-47ce3a4fb022-kube-api-access-h98kg\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.784390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.784361 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38865718-c2c2-48c3-aa91-47ce3a4fb022-tls-certs\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.784880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.784855 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.786458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.786441 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.791864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.791846 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vj9h\" (UniqueName: \"kubernetes.io/projected/daa04960-f591-4f15-8c63-07e571e77dbb-kube-api-access-9vj9h\") pod \"maas-api-revoke-sub-48e07fc0-rww9f\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.888648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.886536 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-model-cache\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.888648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.886708 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-dshm\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.888648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.886787 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-tmp-dir\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.888648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.886974 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-home\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.888648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.887056 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98kg\" (UniqueName: \"kubernetes.io/projected/38865718-c2c2-48c3-aa91-47ce3a4fb022-kube-api-access-h98kg\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.888648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.887125 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38865718-c2c2-48c3-aa91-47ce3a4fb022-tls-certs\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.888648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.888106 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-tmp-dir\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.888648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.888374 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-model-cache\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.889308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.889058 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-home\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.893991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.893720 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:03.898286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.898261 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38865718-c2c2-48c3-aa91-47ce3a4fb022-tls-certs\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.912150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.912055 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-dshm\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.912664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.912634 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98kg\" (UniqueName: \"kubernetes.io/projected/38865718-c2c2-48c3-aa91-47ce3a4fb022-kube-api-access-h98kg\") pod \"rate-test-model-734859-kserve-5b7d566745-wjgrn\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:03.987383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:03.986901 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:04.171566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:04.171475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:04.195738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:04.195710 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f"] Nov 01 16:23:04.196135 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:23:04.196114 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa04960_f591_4f15_8c63_07e571e77dbb.slice/crio-82fd9d9adda1c226e01140d4628b0803ab0e92d5f24c7293f3ea8bb68856e3a5 WatchSource:0}: Error finding container 82fd9d9adda1c226e01140d4628b0803ab0e92d5f24c7293f3ea8bb68856e3a5: Status 404 returned error can't find the container with id 82fd9d9adda1c226e01140d4628b0803ab0e92d5f24c7293f3ea8bb68856e3a5 Nov 01 16:23:04.301701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:04.301675 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn"] Nov 01 16:23:04.364637 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:23:04.364605 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38865718_c2c2_48c3_aa91_47ce3a4fb022.slice/crio-0458f5978b52fb7a9f8f609b5bd656fb9ceb67601d8f234dffe1fc7791ce2ae8 WatchSource:0}: Error finding container 0458f5978b52fb7a9f8f609b5bd656fb9ceb67601d8f234dffe1fc7791ce2ae8: Status 404 returned error can't find the container with id 0458f5978b52fb7a9f8f609b5bd656fb9ceb67601d8f234dffe1fc7791ce2ae8 Nov 01 16:23:04.587156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:04.587120 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:05.144249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:05.144214 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" event={"ID":"38865718-c2c2-48c3-aa91-47ce3a4fb022","Type":"ContainerStarted","Data":"c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1"} Nov 01 16:23:05.144459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:05.144266 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" event={"ID":"38865718-c2c2-48c3-aa91-47ce3a4fb022","Type":"ContainerStarted","Data":"0458f5978b52fb7a9f8f609b5bd656fb9ceb67601d8f234dffe1fc7791ce2ae8"} Nov 01 16:23:05.145908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:05.145884 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" event={"ID":"daa04960-f591-4f15-8c63-07e571e77dbb","Type":"ContainerStarted","Data":"ca61cfd01af812ef6b13950b59907584c67d54d17fee877442e783d3afcbd260"} Nov 01 16:23:05.146012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:05.145925 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" event={"ID":"daa04960-f591-4f15-8c63-07e571e77dbb","Type":"ContainerStarted","Data":"82fd9d9adda1c226e01140d4628b0803ab0e92d5f24c7293f3ea8bb68856e3a5"} Nov 01 16:23:05.163049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:05.162888 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" podStartSLOduration=2.162876613 podStartE2EDuration="2.162876613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:05.162625091 +0000 UTC m=+2122.865791039" watchObservedRunningTime="2026-11-01 16:23:05.162876613 +0000 UTC m=+2122.866042548" Nov 01 16:23:05.181276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:05.181232 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" podStartSLOduration=2.181219745 podStartE2EDuration="2.181219745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:05.180256866 +0000 UTC m=+2122.883422803" watchObservedRunningTime="2026-11-01 16:23:05.181219745 +0000 UTC m=+2122.884385681" Nov 01 16:23:05.433566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:05.433471 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:05.987505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:05.987478 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:06.156124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:06.156087 4573 generic.go:356] "Generic (PLEG): container finished" podID="daa04960-f591-4f15-8c63-07e571e77dbb" containerID="ca61cfd01af812ef6b13950b59907584c67d54d17fee877442e783d3afcbd260" exitCode=0 Nov 01 16:23:06.156308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:06.156176 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" event={"ID":"daa04960-f591-4f15-8c63-07e571e77dbb","Type":"ContainerDied","Data":"ca61cfd01af812ef6b13950b59907584c67d54d17fee877442e783d3afcbd260"} Nov 01 16:23:06.374199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:06.374126 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:07.161869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.161826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:07.364211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.364187 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:07.449729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.449650 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa04960-f591-4f15-8c63-07e571e77dbb-kube-api-access-9vj9h\" (UniqueName: \"kubernetes.io/projected/daa04960-f591-4f15-8c63-07e571e77dbb-kube-api-access-9vj9h\") pod \"daa04960-f591-4f15-8c63-07e571e77dbb\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " Nov 01 16:23:07.449914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.449796 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-xks-ca\") pod \"daa04960-f591-4f15-8c63-07e571e77dbb\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " Nov 01 16:23:07.449914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.449893 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-openshift-ca\") pod \"daa04960-f591-4f15-8c63-07e571e77dbb\" (UID: \"daa04960-f591-4f15-8c63-07e571e77dbb\") " Nov 01 16:23:07.450312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.450284 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-openshift-ca" pod "daa04960-f591-4f15-8c63-07e571e77dbb" (UID: "daa04960-f591-4f15-8c63-07e571e77dbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:23:07.450618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.450597 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:07.451927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.451894 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-xks-ca" pod "daa04960-f591-4f15-8c63-07e571e77dbb" (UID: "daa04960-f591-4f15-8c63-07e571e77dbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:23:07.452116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.452096 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa04960-f591-4f15-8c63-07e571e77dbb-kube-api-access-9vj9h" pod "daa04960-f591-4f15-8c63-07e571e77dbb" (UID: "daa04960-f591-4f15-8c63-07e571e77dbb"). InnerVolumeSpecName "kube-api-access-9vj9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:23:07.552042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.552010 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vj9h\" (UniqueName: \"kubernetes.io/projected/daa04960-f591-4f15-8c63-07e571e77dbb-kube-api-access-9vj9h\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:07.552042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.552038 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daa04960-f591-4f15-8c63-07e571e77dbb-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:07.987738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:07.987704 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:08.174931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:08.174901 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" Nov 01 16:23:08.175314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:08.174925 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f" event={"ID":"daa04960-f591-4f15-8c63-07e571e77dbb","Type":"ContainerDied","Data":"82fd9d9adda1c226e01140d4628b0803ab0e92d5f24c7293f3ea8bb68856e3a5"} Nov 01 16:23:08.175314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:08.174974 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82fd9d9adda1c226e01140d4628b0803ab0e92d5f24c7293f3ea8bb68856e3a5" Nov 01 16:23:08.678742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:08.678702 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:09.344170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:09.344138 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:10.023114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:10.023071 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px"] Nov 01 16:23:10.027497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:10.027469 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfcf586d-z77px"] Nov 01 16:23:10.155635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:10.155603 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:10.459334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:10.459304 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69c5e879-c69f-4c29-b91e-aa7d7cbbb06a" path="/var/lib/kubelet/pods/69c5e879-c69f-4c29-b91e-aa7d7cbbb06a/volumes" Nov 01 16:23:11.182990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:11.182956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:11.952003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:11.951970 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:12.753117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:12.753085 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:13.561433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:13.561402 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:13.988300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:13.988270 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:14.001929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:14.001900 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:14.242064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:14.241986 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:23:14.369166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:14.369121 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:15.080197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:15.080151 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:15.857710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:15.857660 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:16.568665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:16.568628 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:17.463498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.463465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:17.915708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.915672 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79"] Nov 01 16:23:17.916860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.916838 4573 cpu_manager.go:414] "containerMap: removing container" podUID="daa04960-f591-4f15-8c63-07e571e77dbb" containerName="revoke-keys" Nov 01 16:23:17.916934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.916862 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa04960-f591-4f15-8c63-07e571e77dbb" containerName="revoke-keys" Nov 01 16:23:17.917133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.917119 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa04960-f591-4f15-8c63-07e571e77dbb" containerName="revoke-keys" Nov 01 16:23:17.922856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.922830 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:17.929614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.929577 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79"] Nov 01 16:23:17.936130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.936103 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-5ccd7b\"/\"openshift-service-ca.crt\"" Nov 01 16:23:17.936270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.936125 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-5ccd7b\"/\"kube-root-ca.crt\"" Nov 01 16:23:17.936270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.936177 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-5ccd7b\"/\"default-dockercfg-r56k4\"" Nov 01 16:23:17.936479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.936436 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-5ccd7b\"/\"auth-test-model-5ccd7b-kserve-self-signed-certs\"" Nov 01 16:23:17.951505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:17.951477 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:18.021500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.021464 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-dshm\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.021836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.021640 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd17200-6b81-4260-9240-9615be94a2f0-tls-certs\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.021836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.021779 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-model-cache\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.022027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.021859 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-home\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.022027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.021916 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-tmp-dir\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.022200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.022026 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djpxc\" (UniqueName: \"kubernetes.io/projected/5dd17200-6b81-4260-9240-9615be94a2f0-kube-api-access-djpxc\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.025544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.025522 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9"] Nov 01 16:23:18.029833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.029815 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5143c0db-wpjs9"] Nov 01 16:23:18.125818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.123735 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-tmp-dir\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.125818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.123842 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpxc\" (UniqueName: \"kubernetes.io/projected/5dd17200-6b81-4260-9240-9615be94a2f0-kube-api-access-djpxc\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.125818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.124032 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-dshm\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.125818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.124133 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd17200-6b81-4260-9240-9615be94a2f0-tls-certs\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.125818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.124213 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-model-cache\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.125818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.124260 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-home\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.125818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.124738 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-home\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.125818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.125085 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-tmp-dir\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.142780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.136287 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-model-cache\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.160783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.159915 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpxc\" (UniqueName: \"kubernetes.io/projected/5dd17200-6b81-4260-9240-9615be94a2f0-kube-api-access-djpxc\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.164772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.163260 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd17200-6b81-4260-9240-9615be94a2f0-tls-certs\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.196855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.186790 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-dshm\") pod \"auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.247737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.247465 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:18.461572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.461491 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a48f6f-a1de-4d89-b3b6-4a0fa00631a9" path="/var/lib/kubelet/pods/18a48f6f-a1de-4d89-b3b6-4a0fa00631a9/volumes" Nov 01 16:23:18.516945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.514551 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:18.556297 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:23:18.556267 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dd17200_6b81_4260_9240_9615be94a2f0.slice/crio-d748d8a7e6d4b3ee52ec94943c7452cc8f5e4053fe67066a2b4bfd2c89365e5d WatchSource:0}: Error finding container d748d8a7e6d4b3ee52ec94943c7452cc8f5e4053fe67066a2b4bfd2c89365e5d: Status 404 returned error can't find the container with id d748d8a7e6d4b3ee52ec94943c7452cc8f5e4053fe67066a2b4bfd2c89365e5d Nov 01 16:23:18.556635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.556615 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79"] Nov 01 16:23:18.895051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:18.895017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:19.271246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.271159 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" event={"ID":"5dd17200-6b81-4260-9240-9615be94a2f0","Type":"ContainerStarted","Data":"c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92"} Nov 01 16:23:19.271246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.271208 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" event={"ID":"5dd17200-6b81-4260-9240-9615be94a2f0","Type":"ContainerStarted","Data":"d748d8a7e6d4b3ee52ec94943c7452cc8f5e4053fe67066a2b4bfd2c89365e5d"} Nov 01 16:23:19.287418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.287370 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" podStartSLOduration=2.287352815 podStartE2EDuration="2.287352815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:19.286978187 +0000 UTC m=+2136.990144124" watchObservedRunningTime="2026-11-01 16:23:19.287352815 +0000 UTC m=+2136.990518752" Nov 01 16:23:19.386993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.386962 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59"] Nov 01 16:23:19.391745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.391726 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.408852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.408329 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:23:19.408852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.408489 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:23:19.432479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.432425 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59"] Nov 01 16:23:19.455862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.455814 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.456156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.455919 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spjxd\" (UniqueName: \"kubernetes.io/projected/476c00ef-8bd0-4f86-8f66-a13182d309f2-kube-api-access-spjxd\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.456265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.456172 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.504586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.504548 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:19.562851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.561366 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.562851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.561606 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.562851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.561691 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spjxd\" (UniqueName: \"kubernetes.io/projected/476c00ef-8bd0-4f86-8f66-a13182d309f2-kube-api-access-spjxd\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.563243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.563208 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.570785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.570738 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.577096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.577060 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spjxd\" (UniqueName: \"kubernetes.io/projected/476c00ef-8bd0-4f86-8f66-a13182d309f2-kube-api-access-spjxd\") pod \"maas-api-revoke-sub-1a6e3b6f-sst59\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:19.777843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:19.777809 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:20.065416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:20.065376 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59"] Nov 01 16:23:20.065526 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:23:20.065436 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476c00ef_8bd0_4f86_8f66_a13182d309f2.slice/crio-431b99ac0634b7538ccd420c59ebb3ea4fcc93714fe978f6260e9331a5a59f70 WatchSource:0}: Error finding container 431b99ac0634b7538ccd420c59ebb3ea4fcc93714fe978f6260e9331a5a59f70: Status 404 returned error can't find the container with id 431b99ac0634b7538ccd420c59ebb3ea4fcc93714fe978f6260e9331a5a59f70 Nov 01 16:23:20.246579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:20.246506 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:20.248521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:20.248499 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:20.280422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:20.280392 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" event={"ID":"476c00ef-8bd0-4f86-8f66-a13182d309f2","Type":"ContainerStarted","Data":"431b99ac0634b7538ccd420c59ebb3ea4fcc93714fe978f6260e9331a5a59f70"} Nov 01 16:23:21.046917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:21.046882 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:21.290461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:21.290383 4573 generic.go:356] "Generic (PLEG): container finished" podID="476c00ef-8bd0-4f86-8f66-a13182d309f2" containerID="3a70d85f21843463b7e9d9629e2920790e47746b65216fa605dcc7776c620a51" exitCode=0 Nov 01 16:23:21.290890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:21.290474 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" event={"ID":"476c00ef-8bd0-4f86-8f66-a13182d309f2","Type":"ContainerDied","Data":"3a70d85f21843463b7e9d9629e2920790e47746b65216fa605dcc7776c620a51"} Nov 01 16:23:21.713981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:21.713943 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:22.501033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.501004 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:22.512775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.512736 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:22.607009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.606975 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476c00ef-8bd0-4f86-8f66-a13182d309f2-kube-api-access-spjxd\" (UniqueName: \"kubernetes.io/projected/476c00ef-8bd0-4f86-8f66-a13182d309f2-kube-api-access-spjxd\") pod \"476c00ef-8bd0-4f86-8f66-a13182d309f2\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " Nov 01 16:23:22.607221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.607087 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-xks-ca\") pod \"476c00ef-8bd0-4f86-8f66-a13182d309f2\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " Nov 01 16:23:22.607293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.607224 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-openshift-ca\") pod \"476c00ef-8bd0-4f86-8f66-a13182d309f2\" (UID: \"476c00ef-8bd0-4f86-8f66-a13182d309f2\") " Nov 01 16:23:22.608209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.608184 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-openshift-ca" pod "476c00ef-8bd0-4f86-8f66-a13182d309f2" (UID: "476c00ef-8bd0-4f86-8f66-a13182d309f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:23:22.611280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.611240 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476c00ef-8bd0-4f86-8f66-a13182d309f2-kube-api-access-spjxd" pod "476c00ef-8bd0-4f86-8f66-a13182d309f2" (UID: "476c00ef-8bd0-4f86-8f66-a13182d309f2"). InnerVolumeSpecName "kube-api-access-spjxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:23:22.616409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.616384 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-xks-ca" pod "476c00ef-8bd0-4f86-8f66-a13182d309f2" (UID: "476c00ef-8bd0-4f86-8f66-a13182d309f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:23:22.709224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.709186 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:22.709224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.709217 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spjxd\" (UniqueName: \"kubernetes.io/projected/476c00ef-8bd0-4f86-8f66-a13182d309f2-kube-api-access-spjxd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:22.709224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:22.709229 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476c00ef-8bd0-4f86-8f66-a13182d309f2-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:23.155868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:23.155836 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:23.310126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:23.310086 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" Nov 01 16:23:23.310126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:23.310118 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59" event={"ID":"476c00ef-8bd0-4f86-8f66-a13182d309f2","Type":"ContainerDied","Data":"431b99ac0634b7538ccd420c59ebb3ea4fcc93714fe978f6260e9331a5a59f70"} Nov 01 16:23:23.310340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:23.310180 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="431b99ac0634b7538ccd420c59ebb3ea4fcc93714fe978f6260e9331a5a59f70" Nov 01 16:23:23.835092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:23.835055 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:24.364220 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:23:24.364182 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:23:24.580234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:24.580197 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:25.363796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:25.363761 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:26.137872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.137831 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:26.354373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.354340 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt"] Nov 01 16:23:26.355454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.355423 4573 cpu_manager.go:414] "containerMap: removing container" podUID="476c00ef-8bd0-4f86-8f66-a13182d309f2" containerName="revoke-keys" Nov 01 16:23:26.355454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.355449 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476c00ef-8bd0-4f86-8f66-a13182d309f2" containerName="revoke-keys" Nov 01 16:23:26.355657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.355594 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="476c00ef-8bd0-4f86-8f66-a13182d309f2" containerName="revoke-keys" Nov 01 16:23:26.359891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.359871 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.376541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.376516 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt"] Nov 01 16:23:26.453238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.453144 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-9a482b-openshift-default-dockercfg-bmbzh\"" Nov 01 16:23:26.481820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.481790 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.481999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.481848 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.481999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.481896 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.481999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.481921 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.482133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.481997 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.482173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.482155 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.482213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.482182 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pghxk\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-kube-api-access-pghxk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.482252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.482221 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.482291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.482273 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.583989 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584040 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pghxk\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-kube-api-access-pghxk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584075 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584110 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584182 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584218 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584277 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584299 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584386 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584458 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584643 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584675 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.584800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.584729 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.585152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.585131 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.586670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.586645 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.586976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.586956 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.591659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.591634 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.591864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.591845 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pghxk\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-kube-api-access-pghxk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:26.682805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:26.682768 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:27.019700 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:23:27.019672 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12add1b5_31ff_4c5c_b33b_cf8ffd361bf0.slice/crio-d0035f70598c4413a5c55c67cb915e9a29e8f373c8531bea309eb65cde854343 WatchSource:0}: Error finding container d0035f70598c4413a5c55c67cb915e9a29e8f373c8531bea309eb65cde854343: Status 404 returned error can't find the container with id d0035f70598c4413a5c55c67cb915e9a29e8f373c8531bea309eb65cde854343 Nov 01 16:23:27.020805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.020770 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt"] Nov 01 16:23:27.023356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.023319 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.023501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.023397 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.023501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.023437 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:27.109939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.109912 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:27.357610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.357565 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" event={"ID":"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0","Type":"ContainerStarted","Data":"ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71"} Nov 01 16:23:27.357856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.357678 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" event={"ID":"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0","Type":"ContainerStarted","Data":"d0035f70598c4413a5c55c67cb915e9a29e8f373c8531bea309eb65cde854343"} Nov 01 16:23:27.377262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.377204 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" podStartSLOduration=1.377187483 podStartE2EDuration="1.377187483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:27.376360982 +0000 UTC m=+2145.079526922" watchObservedRunningTime="2026-11-01 16:23:27.377187483 +0000 UTC m=+2145.080353420" Nov 01 16:23:27.683914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.683829 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:27.790507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:27.790476 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:28.248591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:28.248561 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:28.287514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:28.287480 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:28.401405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:28.401372 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:23:28.489375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:28.489268 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:28.688568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:28.688532 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 16:23:28.688850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:28.688607 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 16:23:29.181772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:29.181591 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:29.687166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:29.687122 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 16:23:29.687687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:29.687190 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 16:23:29.955500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:29.955378 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:30.684776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:30.684161 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 16:23:30.684776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:30.684233 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 16:23:30.688091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:30.688065 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:31.519451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:31.519416 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:31.686777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:31.686291 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 16:23:31.686777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:31.686372 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 16:23:32.059320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.059287 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:32.405493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.405456 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:32.547579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.547546 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:32.628325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.626497 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95"] Nov 01 16:23:32.632434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.632407 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.637638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.637609 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95"] Nov 01 16:23:32.644261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.643367 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:23:32.645793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.644459 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:23:32.694833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.689180 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" start-of-body= Nov 01 16:23:32.694833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.689238 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.238:15021/healthz/ready\": dial tcp 10.133.0.238:15021: connect: connection refused" Nov 01 16:23:32.723990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.723952 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.724211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.724091 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv25t\" (UniqueName: \"kubernetes.io/projected/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-kube-api-access-hv25t\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.724296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.724256 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.826169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.826129 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.826351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.826284 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv25t\" (UniqueName: \"kubernetes.io/projected/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-kube-api-access-hv25t\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.826475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.826457 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.827160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.827139 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.829361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.829338 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.835580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.835559 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv25t\" (UniqueName: \"kubernetes.io/projected/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-kube-api-access-hv25t\") pod \"maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:32.958836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:32.958744 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:33.282348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:33.282314 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95"] Nov 01 16:23:33.439089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:33.439057 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" event={"ID":"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0","Type":"ContainerStarted","Data":"4afed5fb620bf2b6066047e32fb29d87104fe75cd408c80a9ae1d2ac6e5f01a3"} Nov 01 16:23:33.471905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:33.471868 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:33.838917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:33.838883 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:33.839507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:33.839481 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:33.840477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:33.840456 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:23:34.299056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:34.299018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:34.463025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:34.462920 4573 generic.go:356] "Generic (PLEG): container finished" podID="ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0" containerID="57ee74ba3924b50682101b5038b8406e519e27c47bf6532712f2d63c426ac966" exitCode=0 Nov 01 16:23:34.481575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:34.481535 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" event={"ID":"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0","Type":"ContainerDied","Data":"57ee74ba3924b50682101b5038b8406e519e27c47bf6532712f2d63c426ac966"} Nov 01 16:23:35.179669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.179619 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:35.643797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.643761 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:35.689770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.689722 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-openshift-ca\") pod \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " Nov 01 16:23:35.689973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.689793 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-kube-api-access-hv25t\" (UniqueName: \"kubernetes.io/projected/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-kube-api-access-hv25t\") pod \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " Nov 01 16:23:35.690094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.690071 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-openshift-ca" pod "ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0" (UID: "ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:23:35.690177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.690158 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-xks-ca\") pod \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\" (UID: \"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0\") " Nov 01 16:23:35.690962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.690935 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:35.692029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.692006 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-xks-ca" pod "ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0" (UID: "ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:23:35.692100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.692088 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-kube-api-access-hv25t" pod "ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0" (UID: "ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0"). InnerVolumeSpecName "kube-api-access-hv25t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:23:35.792046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.791976 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv25t\" (UniqueName: \"kubernetes.io/projected/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-kube-api-access-hv25t\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:35.792046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:35.792003 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:23:36.005300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:36.005268 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:36.480874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:36.480848 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" Nov 01 16:23:36.481034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:36.480887 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95" event={"ID":"ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0","Type":"ContainerDied","Data":"4afed5fb620bf2b6066047e32fb29d87104fe75cd408c80a9ae1d2ac6e5f01a3"} Nov 01 16:23:36.481034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:36.480934 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4afed5fb620bf2b6066047e32fb29d87104fe75cd408c80a9ae1d2ac6e5f01a3" Nov 01 16:23:36.770731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:36.770649 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:37.021649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:37.021569 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7"] Nov 01 16:23:37.026247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:37.026218 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6213a38-n7zv7"] Nov 01 16:23:37.670816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:37.670778 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:38.462993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:38.462958 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dea3b94-4699-48b5-93ab-ad2c031557d0" path="/var/lib/kubelet/pods/8dea3b94-4699-48b5-93ab-ad2c031557d0/volumes" Nov 01 16:23:38.486280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:38.486247 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:39.362814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:39.362777 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:40.149740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:40.149685 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:40.947322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:40.947291 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:41.387165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.387131 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747"] Nov 01 16:23:41.387918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.387893 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0" containerName="revoke-keys" Nov 01 16:23:41.387918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.387916 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0" containerName="revoke-keys" Nov 01 16:23:41.388095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.388081 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0" containerName="revoke-keys" Nov 01 16:23:41.393347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.393320 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.402916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.402890 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:23:41.520796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.516876 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747"] Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.543570 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.543613 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6721a2-bcda-4eab-a726-90812e0dfb06-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.543724 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.543791 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.543881 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.543997 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.544072 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlvxk\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-kube-api-access-nlvxk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.544198 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.549214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.544256 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646094 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646160 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646246 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646273 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6721a2-bcda-4eab-a726-90812e0dfb06-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646360 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646408 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646478 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646590 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646630 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646636 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlvxk\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-kube-api-access-nlvxk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.646719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.646665 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.647071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.647049 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.647154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.647135 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.647549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.647533 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6721a2-bcda-4eab-a726-90812e0dfb06-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.649079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.649049 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.649206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.649145 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.654721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.654693 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.654894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.654746 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlvxk\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-kube-api-access-nlvxk\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.855162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.855122 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:41.898277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:41.898201 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:42.154901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.154869 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747"] Nov 01 16:23:42.155049 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:23:42.154931 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6721a2_bcda_4eab_a726_90812e0dfb06.slice/crio-08c7bc1a06c06a364920d0ef795a143714730eed6fd958d4858b46f31af5acb5 WatchSource:0}: Error finding container 08c7bc1a06c06a364920d0ef795a143714730eed6fd958d4858b46f31af5acb5: Status 404 returned error can't find the container with id 08c7bc1a06c06a364920d0ef795a143714730eed6fd958d4858b46f31af5acb5 Nov 01 16:23:42.158360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.158327 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.158444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.158406 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.158483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.158449 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:42.565018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.564933 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" event={"ID":"7e6721a2-bcda-4eab-a726-90812e0dfb06","Type":"ContainerStarted","Data":"6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b"} Nov 01 16:23:42.565393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.565024 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" event={"ID":"7e6721a2-bcda-4eab-a726-90812e0dfb06","Type":"ContainerStarted","Data":"08c7bc1a06c06a364920d0ef795a143714730eed6fd958d4858b46f31af5acb5"} Nov 01 16:23:42.590615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.590549 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" podStartSLOduration=1.5905298559999999 podStartE2EDuration="1.590529856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:42.586269267 +0000 UTC m=+2160.289435205" watchObservedRunningTime="2026-11-01 16:23:42.590529856 +0000 UTC m=+2160.293695794" Nov 01 16:23:42.718220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.718181 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw"] Nov 01 16:23:42.787614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.786608 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:42.855493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.855237 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:42.885779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:42.885653 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:43.005644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:43.005591 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:43.873774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:43.866869 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 01 16:23:43.873774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:43.866948 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 01 16:23:44.769505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:44.769438 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:44.856570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:44.856535 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 01 16:23:44.856833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:44.856592 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 01 16:23:45.856329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:45.856294 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 01 16:23:45.856863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:45.856358 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 01 16:23:45.873775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:45.861897 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:46.855542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:46.855507 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 01 16:23:46.855736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:46.855566 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 01 16:23:46.903316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:46.903280 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:47.691573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:47.691537 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:47.718530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:47.718498 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" podUID="88acad1c-61c1-4887-8abc-767ccf45e509" containerName="payload-processing" containerID="cri-o://ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d" gracePeriod=25 Nov 01 16:23:47.862776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:47.855850 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" start-of-body= Nov 01 16:23:47.862776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:47.855910 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.240:15021/healthz/ready\": dial tcp 10.133.0.240:15021: connect: connection refused" Nov 01 16:23:48.425823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:48.425768 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:48.932465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:48.932430 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:48.932939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:48.932912 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:48.933843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:48.933813 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:23:49.450126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:49.450090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:50.181255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:50.181217 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:50.948274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:50.948241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:51.670645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:51.670608 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:52.468350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:52.468317 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:53.149476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:53.149437 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:53.808239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:53.808208 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:54.025458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:54.025422 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" podUID="88acad1c-61c1-4887-8abc-767ccf45e509" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:23:54.367677 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:23:54.367634 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:23:54.531391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:54.531363 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:55.028057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:55.028022 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88"] Nov 01 16:23:55.032418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:55.032391 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09a26172-f7k88"] Nov 01 16:23:55.383717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:55.383678 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:56.065420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.065381 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:56.387824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.387791 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524"] Nov 01 16:23:56.394413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.394386 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.406662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.406635 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524"] Nov 01 16:23:56.464714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.464682 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72f3e0c-6521-441d-8c54-850fb767d5d8" path="/var/lib/kubelet/pods/f72f3e0c-6521-441d-8c54-850fb767d5d8/volumes" Nov 01 16:23:56.491520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.491485 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.491681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.491560 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.491681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.491670 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.491791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.491738 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.491890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.491868 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgbgb\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-kube-api-access-bgbgb\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.491933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.491911 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.491933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.491928 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.492062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.492047 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.492108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.492073 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593132 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593185 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593242 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593298 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593450 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgbgb\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-kube-api-access-bgbgb\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593502 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593528 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-credential-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593578 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-socket\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593620 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593699 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593627 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-certs\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593729 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.593953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.593817 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-data\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.594337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.594316 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istiod-ca-cert\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.595852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.595827 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-envoy\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.595963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.595947 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-podinfo\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.601659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.601631 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-token\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.601838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.601822 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgbgb\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-kube-api-access-bgbgb\") pod \"e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.715914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.715833 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:56.749174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:56.749141 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:57.007472 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:23:57.007441 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3ec6a5_f442_4335_b93c_b6bcea7de99f.slice/crio-6afdc96535e7d42a0889208ef6bee4ce6426bf0631d428c5a7044b6ed284fef0 WatchSource:0}: Error finding container 6afdc96535e7d42a0889208ef6bee4ce6426bf0631d428c5a7044b6ed284fef0: Status 404 returned error can't find the container with id 6afdc96535e7d42a0889208ef6bee4ce6426bf0631d428c5a7044b6ed284fef0 Nov 01 16:23:57.009047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.009022 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524"] Nov 01 16:23:57.010665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.010631 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:57.010796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.010711 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:57.010878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.010847 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:23:57.034664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.034637 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c"] Nov 01 16:23:57.041349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.041324 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18437e32-x5q9c"] Nov 01 16:23:57.444576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.444536 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:57.735872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.735783 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" event={"ID":"1c3ec6a5-f442-4335-b93c-b6bcea7de99f","Type":"ContainerStarted","Data":"db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6"} Nov 01 16:23:57.736040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.735875 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" event={"ID":"1c3ec6a5-f442-4335-b93c-b6bcea7de99f","Type":"ContainerStarted","Data":"6afdc96535e7d42a0889208ef6bee4ce6426bf0631d428c5a7044b6ed284fef0"} Nov 01 16:23:57.760331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:57.760283 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" podStartSLOduration=1.760270589 podStartE2EDuration="1.760270589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:23:57.754641065 +0000 UTC m=+2175.457807037" watchObservedRunningTime="2026-11-01 16:23:57.760270589 +0000 UTC m=+2175.463436553" Nov 01 16:23:58.183783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:58.182244 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:58.460179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:58.460095 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b50cf165-0412-4978-9947-392132261849" path="/var/lib/kubelet/pods/b50cf165-0412-4978-9947-392132261849/volumes" Nov 01 16:23:58.716632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:58.716546 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:23:58.718075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:58.718050 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 16:23:58.718522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:58.718112 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 16:23:58.913022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:58.912989 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:59.451902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:59.451854 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:59.603031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:59.602995 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:59.671739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:59.671695 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95"] Nov 01 16:23:59.730832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:59.726870 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 16:23:59.730832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:59.726936 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 16:23:59.730832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:59.727726 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:23:59.747773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:59.746144 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-9e044c-9cm95"] Nov 01 16:23:59.810075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:23:59.805678 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:00.102715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:00.102626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:00.459431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:00.459401 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0" path="/var/lib/kubelet/pods/ba9983cf-d1ab-4314-ba67-3c5fa4dc82a0/volumes" Nov 01 16:24:00.716424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:00.716343 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 16:24:00.716424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:00.716407 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 16:24:00.920697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:00.920665 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:01.055728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.055652 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk"] Nov 01 16:24:01.064101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.064069 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.100786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.099076 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9e044c-openshift-default-dockercfg-4tjb7\"" Nov 01 16:24:01.102649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.102618 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk"] Nov 01 16:24:01.130325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130294 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.130497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130417 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.130497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130460 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.130587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130506 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.130643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130629 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.130707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130691 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fc2fe9-898e-4e4f-92f9-e71133689886-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.130860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130839 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s6gq\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-kube-api-access-8s6gq\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.130936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130923 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.130974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.130967 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.232978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.232943 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fc2fe9-898e-4e4f-92f9-e71133689886-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233045 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s6gq\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-kube-api-access-8s6gq\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233099 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233149 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233191 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233250 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233274 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233320 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233374 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233547 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-certs\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233641 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233829 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fc2fe9-898e-4e4f-92f9-e71133689886-istiod-ca-cert\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.233923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.233876 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-data\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.234060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.234042 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-credential-socket\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.235669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.235644 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-envoy\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.236211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.236187 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-podinfo\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.240336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.240309 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-token\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.240537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.240520 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s6gq\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-kube-api-access-8s6gq\") pod \"e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.415490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.415457 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:01.588863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.573302 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:01.716462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.716360 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 16:24:01.716462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.716422 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 16:24:01.737931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.737902 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk"] Nov 01 16:24:01.738689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.738639 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:01.738818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.738798 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:01.738900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.738884 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:24:01.837878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:01.837842 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" event={"ID":"c0fc2fe9-898e-4e4f-92f9-e71133689886","Type":"ContainerStarted","Data":"66a77e7d9a35442e07669810f66867dec33004fd05fa9076718c8236ee5e66bf"} Nov 01 16:24:02.388021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:02.387990 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:02.716830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:02.716686 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Nov 01 16:24:02.716830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:02.716789 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Nov 01 16:24:02.848537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:02.848502 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" event={"ID":"c0fc2fe9-898e-4e4f-92f9-e71133689886","Type":"ContainerStarted","Data":"969ab3e75d7ece6512777dfff878a8d43e34ef1f4780eae1ff0e530f03c8890c"} Nov 01 16:24:02.906774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:02.906573 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" podStartSLOduration=1.906539838 podStartE2EDuration="1.906539838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:24:02.884485074 +0000 UTC m=+2180.587651040" watchObservedRunningTime="2026-11-01 16:24:02.906539838 +0000 UTC m=+2180.609705775" Nov 01 16:24:03.175778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:03.175272 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:03.415826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:03.415788 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:03.417278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:03.417173 4573 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 16:24:03.417278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:03.417238 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 16:24:03.899650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:03.898649 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:24:03.899650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:03.899591 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:24:03.900508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:03.900479 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:24:03.982778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:03.982387 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:04.026208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:04.026173 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" podUID="88acad1c-61c1-4887-8abc-767ccf45e509" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:24:04.416062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:04.416026 4573 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 16:24:04.416245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:04.416080 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 16:24:04.719448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:04.719368 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:05.386807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:05.386226 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:05.416560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:05.416530 4573 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 16:24:05.416712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:05.416584 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 16:24:06.026046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:06.026010 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552"] Nov 01 16:24:06.031824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:06.031231 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0934582b-pl552"] Nov 01 16:24:06.187407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:06.187358 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:06.363839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:06.344571 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:06.417405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:06.417372 4573 patch_prober.go:29] interesting pod/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Nov 01 16:24:06.418131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:06.418097 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Nov 01 16:24:06.842233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:06.840537 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:07.040228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.040196 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13eeb92-2eeb-44bb-a30b-3c6843e8ffad" path="/var/lib/kubelet/pods/a13eeb92-2eeb-44bb-a30b-3c6843e8ffad/volumes" Nov 01 16:24:07.040662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.040642 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk"] Nov 01 16:24:07.040806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.040707 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77"] Nov 01 16:24:07.041071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.041041 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerName="istio-proxy" containerID="cri-o://969ab3e75d7ece6512777dfff878a8d43e34ef1f4780eae1ff0e530f03c8890c" gracePeriod=30 Nov 01 16:24:07.046987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.046966 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77"] Nov 01 16:24:07.047110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.047054 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.056791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.056326 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:24:07.056791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.056409 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:24:07.232300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.232268 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2b9l\" (UniqueName: \"kubernetes.io/projected/d2459b92-5db6-4cd7-857e-ae74d138cce2-kube-api-access-t2b9l\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.232523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.232353 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.232523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.232460 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.334277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.334237 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2b9l\" (UniqueName: \"kubernetes.io/projected/d2459b92-5db6-4cd7-857e-ae74d138cce2-kube-api-access-t2b9l\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.334459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.334350 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.334459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.334402 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.335036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.335008 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.337039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.337015 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.342266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.342247 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2b9l\" (UniqueName: \"kubernetes.io/projected/d2459b92-5db6-4cd7-857e-ae74d138cce2-kube-api-access-t2b9l\") pod \"maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.380773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.370801 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:07.386781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.385012 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:07.743711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:07.743684 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77"] Nov 01 16:24:07.744091 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:24:07.744023 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2459b92_5db6_4cd7_857e_ae74d138cce2.slice/crio-6756b0a9ca5fb8e307e83be34c5034e8b85f4b7f699df00c1860ff7a19d36cfa WatchSource:0}: Error finding container 6756b0a9ca5fb8e307e83be34c5034e8b85f4b7f699df00c1860ff7a19d36cfa: Status 404 returned error can't find the container with id 6756b0a9ca5fb8e307e83be34c5034e8b85f4b7f699df00c1860ff7a19d36cfa Nov 01 16:24:08.025228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.025198 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:24:08.053672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.050726 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88acad1c-61c1-4887-8abc-767ccf45e509-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acad1c-61c1-4887-8abc-767ccf45e509-plugins-config-volume\") pod \"88acad1c-61c1-4887-8abc-767ccf45e509\" (UID: \"88acad1c-61c1-4887-8abc-767ccf45e509\") " Nov 01 16:24:08.053672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.050797 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88acad1c-61c1-4887-8abc-767ccf45e509-kube-api-access-tf9zd\" (UniqueName: \"kubernetes.io/projected/88acad1c-61c1-4887-8abc-767ccf45e509-kube-api-access-tf9zd\") pod \"88acad1c-61c1-4887-8abc-767ccf45e509\" (UID: \"88acad1c-61c1-4887-8abc-767ccf45e509\") " Nov 01 16:24:08.053672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.051921 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88acad1c-61c1-4887-8abc-767ccf45e509-plugins-config-volume" pod "88acad1c-61c1-4887-8abc-767ccf45e509" (UID: "88acad1c-61c1-4887-8abc-767ccf45e509"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:08.055909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.055882 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88acad1c-61c1-4887-8abc-767ccf45e509-kube-api-access-tf9zd" pod "88acad1c-61c1-4887-8abc-767ccf45e509" (UID: "88acad1c-61c1-4887-8abc-767ccf45e509"). InnerVolumeSpecName "kube-api-access-tf9zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:08.074850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.072170 4573 generic.go:356] "Generic (PLEG): container finished" podID="88acad1c-61c1-4887-8abc-767ccf45e509" containerID="ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d" exitCode=0 Nov 01 16:24:08.074850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.072386 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" event={"ID":"88acad1c-61c1-4887-8abc-767ccf45e509","Type":"ContainerDied","Data":"ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d"} Nov 01 16:24:08.074850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.072442 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" event={"ID":"88acad1c-61c1-4887-8abc-767ccf45e509","Type":"ContainerDied","Data":"e811ea7fa4ddf0c0230084a0f01d072d00029a9ddfe1ea4d0b9737a0f9315f37"} Nov 01 16:24:08.074850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.072473 4573 scope.go:122] "RemoveContainer" containerID="ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d" Nov 01 16:24:08.074850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.072629 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw" Nov 01 16:24:08.122433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.122393 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" event={"ID":"d2459b92-5db6-4cd7-857e-ae74d138cce2","Type":"ContainerStarted","Data":"6756b0a9ca5fb8e307e83be34c5034e8b85f4b7f699df00c1860ff7a19d36cfa"} Nov 01 16:24:08.143977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.143134 4573 scope.go:122] "RemoveContainer" containerID="ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d" Nov 01 16:24:08.145227 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:24:08.145171 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d\": container with ID starting with ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d not found: ID does not exist" containerID="ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d" Nov 01 16:24:08.145344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.145239 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d"} err="failed to get container status \"ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d\": rpc error: code = NotFound desc = could not find container \"ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d\": container with ID starting with ef973d6495f93d2aa764d1e9ae58b84339bf4c0280a20d179d29abc1fff0be1d not found: ID does not exist" Nov 01 16:24:08.151619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.151596 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw"] Nov 01 16:24:08.153197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.153180 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88acad1c-61c1-4887-8abc-767ccf45e509-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:08.153316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.153305 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9zd\" (UniqueName: \"kubernetes.io/projected/88acad1c-61c1-4887-8abc-767ccf45e509-kube-api-access-tf9zd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:08.155964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.155945 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-9e044c-75ffc66cdc-d5bnw"] Nov 01 16:24:08.254147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.254116 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:08.477275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.477238 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88acad1c-61c1-4887-8abc-767ccf45e509" path="/var/lib/kubelet/pods/88acad1c-61c1-4887-8abc-767ccf45e509/volumes" Nov 01 16:24:08.966848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:08.966810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:09.135983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:09.135939 4573 generic.go:356] "Generic (PLEG): container finished" podID="d2459b92-5db6-4cd7-857e-ae74d138cce2" containerID="4405967305bd31d106beacca517c1322f7730e456cb6af20b18b27576f827b39" exitCode=0 Nov 01 16:24:09.136184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:09.135999 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" event={"ID":"d2459b92-5db6-4cd7-857e-ae74d138cce2","Type":"ContainerDied","Data":"4405967305bd31d106beacca517c1322f7730e456cb6af20b18b27576f827b39"} Nov 01 16:24:09.757775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:09.755290 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:10.310684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.310662 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:10.389560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.389529 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:10.401464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.401398 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-xks-ca\") pod \"d2459b92-5db6-4cd7-857e-ae74d138cce2\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " Nov 01 16:24:10.401464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.401442 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-openshift-ca\") pod \"d2459b92-5db6-4cd7-857e-ae74d138cce2\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " Nov 01 16:24:10.401650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.401522 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2459b92-5db6-4cd7-857e-ae74d138cce2-kube-api-access-t2b9l\" (UniqueName: \"kubernetes.io/projected/d2459b92-5db6-4cd7-857e-ae74d138cce2-kube-api-access-t2b9l\") pod \"d2459b92-5db6-4cd7-857e-ae74d138cce2\" (UID: \"d2459b92-5db6-4cd7-857e-ae74d138cce2\") " Nov 01 16:24:10.402120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.402094 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-openshift-ca" pod "d2459b92-5db6-4cd7-857e-ae74d138cce2" (UID: "d2459b92-5db6-4cd7-857e-ae74d138cce2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:10.403889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.403865 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-xks-ca" pod "d2459b92-5db6-4cd7-857e-ae74d138cce2" (UID: "d2459b92-5db6-4cd7-857e-ae74d138cce2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:24:10.404164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.404134 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2459b92-5db6-4cd7-857e-ae74d138cce2-kube-api-access-t2b9l" pod "d2459b92-5db6-4cd7-857e-ae74d138cce2" (UID: "d2459b92-5db6-4cd7-857e-ae74d138cce2"). InnerVolumeSpecName "kube-api-access-t2b9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:10.503931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.503894 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2b9l\" (UniqueName: \"kubernetes.io/projected/d2459b92-5db6-4cd7-857e-ae74d138cce2-kube-api-access-t2b9l\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:10.503931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.503934 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:10.504144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:10.503949 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2459b92-5db6-4cd7-857e-ae74d138cce2-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:11.154375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:11.154343 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" Nov 01 16:24:11.154547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:11.154345 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77" event={"ID":"d2459b92-5db6-4cd7-857e-ae74d138cce2","Type":"ContainerDied","Data":"6756b0a9ca5fb8e307e83be34c5034e8b85f4b7f699df00c1860ff7a19d36cfa"} Nov 01 16:24:11.154547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:11.154481 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6756b0a9ca5fb8e307e83be34c5034e8b85f4b7f699df00c1860ff7a19d36cfa" Nov 01 16:24:11.182320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:11.182289 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:11.969928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:11.969891 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:12.620685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:12.620651 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:13.467274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:13.467244 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:14.025057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.024972 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5"] Nov 01 16:24:14.029211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.029180 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6590b71-82xz5"] Nov 01 16:24:14.181641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.181610 4573 generic.go:356] "Generic (PLEG): container finished" podID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerID="969ab3e75d7ece6512777dfff878a8d43e34ef1f4780eae1ff0e530f03c8890c" exitCode=0 Nov 01 16:24:14.181843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.181686 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" event={"ID":"c0fc2fe9-898e-4e4f-92f9-e71133689886","Type":"ContainerDied","Data":"969ab3e75d7ece6512777dfff878a8d43e34ef1f4780eae1ff0e530f03c8890c"} Nov 01 16:24:14.185493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.185471 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:14.449105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.449081 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:14.460118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.460094 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fdcd16e-5143-4a27-8e53-c11f75e1e99a" path="/var/lib/kubelet/pods/6fdcd16e-5143-4a27-8e53-c11f75e1e99a/volumes" Nov 01 16:24:14.573385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573357 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-data\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.573907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573422 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-envoy\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.573907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573487 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-kube-api-access-8s6gq\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-kube-api-access-8s6gq\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.573907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573625 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-data" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:14.573907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573647 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-token\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-token\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.573907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573687 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-certs\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.573907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573731 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-credential-socket\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.573907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573847 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fc2fe9-898e-4e4f-92f9-e71133689886-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fc2fe9-898e-4e4f-92f9-e71133689886-istiod-ca-cert\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.573907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573899 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-socket\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.574380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.573931 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-podinfo\") pod \"c0fc2fe9-898e-4e4f-92f9-e71133689886\" (UID: \"c0fc2fe9-898e-4e4f-92f9-e71133689886\") " Nov 01 16:24:14.574380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.574028 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-certs" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:14.574380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.574331 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fc2fe9-898e-4e4f-92f9-e71133689886-istiod-ca-cert" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:14.574711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.574688 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.575089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.575020 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fc2fe9-898e-4e4f-92f9-e71133689886-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.575089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.575045 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.575089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.574836 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-socket" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:14.575089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.574975 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-credential-socket" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:14.576002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.575964 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-envoy" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:14.576106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.576084 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-token" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:14.576211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.576190 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-kube-api-access-8s6gq" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "kube-api-access-8s6gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:14.576367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.576349 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-podinfo" pod "c0fc2fe9-898e-4e4f-92f9-e71133689886" (UID: "c0fc2fe9-898e-4e4f-92f9-e71133689886"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:24:14.676922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.676894 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.676922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.676920 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s6gq\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-kube-api-access-8s6gq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.677088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.676932 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.677088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.676941 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.677088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.676949 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fc2fe9-898e-4e4f-92f9-e71133689886-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.677088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.676958 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fc2fe9-898e-4e4f-92f9-e71133689886-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:14.830518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:14.830483 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:15.191709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:15.191675 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" Nov 01 16:24:15.191953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:15.191705 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk" event={"ID":"c0fc2fe9-898e-4e4f-92f9-e71133689886","Type":"ContainerDied","Data":"66a77e7d9a35442e07669810f66867dec33004fd05fa9076718c8236ee5e66bf"} Nov 01 16:24:15.191953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:15.191807 4573 scope.go:122] "RemoveContainer" containerID="969ab3e75d7ece6512777dfff878a8d43e34ef1f4780eae1ff0e530f03c8890c" Nov 01 16:24:15.216072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:15.216046 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk"] Nov 01 16:24:15.221509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:15.221488 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9e044c-openshift-default-85757fb4b9-tj5xk"] Nov 01 16:24:15.490121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:15.490038 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:15.848531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:15.848451 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws"] Nov 01 16:24:15.848959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:15.848661 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" containerID="cri-o://c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3" gracePeriod=30 Nov 01 16:24:16.178554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:16.178519 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:16.461909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:16.461832 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" path="/var/lib/kubelet/pods/c0fc2fe9-898e-4e4f-92f9-e71133689886/volumes" Nov 01 16:24:17.039248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:17.039215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:17.133213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:17.133165 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2"] Nov 01 16:24:17.216815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:17.216145 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:17.306460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:17.306355 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:17.862203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:17.862169 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:18.537598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:18.537569 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:19.188453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:19.188420 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:19.879054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:19.879023 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:20.368767 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:20.368729 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:20.434312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:20.434282 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:20.565871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:20.565842 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:21.136172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.136116 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 16:24:21.206595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.206556 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-podinfo\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.206890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.206625 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-certs\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.207193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.206740 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59186df5-07c5-4dbc-b45d-d474e1514fe6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59186df5-07c5-4dbc-b45d-d474e1514fe6-istiod-ca-cert\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.207309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.206980 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-certs" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:21.207309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.207156 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59186df5-07c5-4dbc-b45d-d474e1514fe6-istiod-ca-cert" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:21.207454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.207428 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-data\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.207533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.207517 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-kube-api-access-546gl\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-kube-api-access-546gl\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.207688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.207575 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-envoy\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.207796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.207665 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-data" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:21.207796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.207709 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-socket\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.207796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.207740 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-credential-socket\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.207986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.207819 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-token\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-token\") pod \"59186df5-07c5-4dbc-b45d-d474e1514fe6\" (UID: \"59186df5-07c5-4dbc-b45d-d474e1514fe6\") " Nov 01 16:24:21.208255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.208225 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-socket" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:21.208684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.208418 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-credential-socket" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:21.208684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.208588 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.208684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.208608 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59186df5-07c5-4dbc-b45d-d474e1514fe6-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.208684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.208625 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.208684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.208639 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.208684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.208652 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.209323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.209297 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-podinfo" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:24:21.210015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.209988 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-kube-api-access-546gl" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "kube-api-access-546gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:21.210676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.210645 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-token" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:21.210805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.210741 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-envoy" pod "59186df5-07c5-4dbc-b45d-d474e1514fe6" (UID: "59186df5-07c5-4dbc-b45d-d474e1514fe6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:24:21.216668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.216618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:21.310296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.310198 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-546gl\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-kube-api-access-546gl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.310296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.310235 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.310296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.310249 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.310296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.310262 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59186df5-07c5-4dbc-b45d-d474e1514fe6-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:21.318583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.318548 4573 generic.go:356] "Generic (PLEG): container finished" podID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerID="c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3" exitCode=0 Nov 01 16:24:21.318801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.318606 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" Nov 01 16:24:21.318801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.318622 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" event={"ID":"59186df5-07c5-4dbc-b45d-d474e1514fe6","Type":"ContainerDied","Data":"c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3"} Nov 01 16:24:21.318801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.318712 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws" event={"ID":"59186df5-07c5-4dbc-b45d-d474e1514fe6","Type":"ContainerDied","Data":"736ded45db8fc1639cb66a356366b767c60b4ff9b901a95dce9e97a034ff0dce"} Nov 01 16:24:21.318801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.318744 4573 scope.go:122] "RemoveContainer" containerID="c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3" Nov 01 16:24:21.335602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.335575 4573 scope.go:122] "RemoveContainer" containerID="36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a" Nov 01 16:24:21.347306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.347279 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws"] Nov 01 16:24:21.347886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.347860 4573 scope.go:122] "RemoveContainer" containerID="c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3" Nov 01 16:24:21.348220 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:24:21.348199 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3\": container with ID starting with c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3 not found: ID does not exist" containerID="c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3" Nov 01 16:24:21.348289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.348229 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3"} err="failed to get container status \"c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3\": rpc error: code = NotFound desc = could not find container \"c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3\": container with ID starting with c3fd1203d3fada21c164129c71ac33c803dbf27553b67685fcd28abf117abca3 not found: ID does not exist" Nov 01 16:24:21.348289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.348252 4573 scope.go:122] "RemoveContainer" containerID="36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a" Nov 01 16:24:21.348529 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:24:21.348508 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a\": container with ID starting with 36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a not found: ID does not exist" containerID="36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a" Nov 01 16:24:21.348580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.348537 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a"} err="failed to get container status \"36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a\": rpc error: code = NotFound desc = could not find container \"36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a\": container with ID starting with 36b617e647d28e89834ee7275f7603f4f395315af17d4b7767fc477431baec3a not found: ID does not exist" Nov 01 16:24:21.351565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.351545 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-7r8ws"] Nov 01 16:24:21.931471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:21.931425 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:22.133822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.133781 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" podUID="45395229-36ee-4bce-9004-e3be3041e165" containerName="payload-processing" containerID="cri-o://7c98164b144899f8984c5c44869c6d88b63ba685b7a28cf6cc435349d3360dfa" gracePeriod=25 Nov 01 16:24:22.346554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.346518 4573 generic.go:356] "Generic (PLEG): container finished" podID="45395229-36ee-4bce-9004-e3be3041e165" containerID="7c98164b144899f8984c5c44869c6d88b63ba685b7a28cf6cc435349d3360dfa" exitCode=0 Nov 01 16:24:22.347016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.346688 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" event={"ID":"45395229-36ee-4bce-9004-e3be3041e165","Type":"ContainerDied","Data":"7c98164b144899f8984c5c44869c6d88b63ba685b7a28cf6cc435349d3360dfa"} Nov 01 16:24:22.455403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.455357 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:24:22.461657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.461631 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" path="/var/lib/kubelet/pods/59186df5-07c5-4dbc-b45d-d474e1514fe6/volumes" Nov 01 16:24:22.526735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.526648 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45395229-36ee-4bce-9004-e3be3041e165-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45395229-36ee-4bce-9004-e3be3041e165-plugins-config-volume\") pod \"45395229-36ee-4bce-9004-e3be3041e165\" (UID: \"45395229-36ee-4bce-9004-e3be3041e165\") " Nov 01 16:24:22.526933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.526740 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45395229-36ee-4bce-9004-e3be3041e165-kube-api-access-mxgrf\" (UniqueName: \"kubernetes.io/projected/45395229-36ee-4bce-9004-e3be3041e165-kube-api-access-mxgrf\") pod \"45395229-36ee-4bce-9004-e3be3041e165\" (UID: \"45395229-36ee-4bce-9004-e3be3041e165\") " Nov 01 16:24:22.527200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.527168 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45395229-36ee-4bce-9004-e3be3041e165-plugins-config-volume" pod "45395229-36ee-4bce-9004-e3be3041e165" (UID: "45395229-36ee-4bce-9004-e3be3041e165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:22.528278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.528250 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45395229-36ee-4bce-9004-e3be3041e165-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:22.530486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.530458 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45395229-36ee-4bce-9004-e3be3041e165-kube-api-access-mxgrf" pod "45395229-36ee-4bce-9004-e3be3041e165" (UID: "45395229-36ee-4bce-9004-e3be3041e165"). InnerVolumeSpecName "kube-api-access-mxgrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:22.629923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.629890 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxgrf\" (UniqueName: \"kubernetes.io/projected/45395229-36ee-4bce-9004-e3be3041e165-kube-api-access-mxgrf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:22.654857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:22.654826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:23.259850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.259804 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5"] Nov 01 16:24:23.260119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.260089 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" podUID="b37c04ce-7216-422c-b973-974e69057427" containerName="maas-api" containerID="cri-o://05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c" gracePeriod=30 Nov 01 16:24:23.316290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.316150 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:23.357382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.357349 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" Nov 01 16:24:23.357869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.357380 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2" event={"ID":"45395229-36ee-4bce-9004-e3be3041e165","Type":"ContainerDied","Data":"b2d2d73908493b5f0d0370acd6d50690430ab56dcf6923ccdf6a732e6d1ef350"} Nov 01 16:24:23.357869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.357450 4573 scope.go:122] "RemoveContainer" containerID="7c98164b144899f8984c5c44869c6d88b63ba685b7a28cf6cc435349d3360dfa" Nov 01 16:24:23.399716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.399685 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2"] Nov 01 16:24:23.410302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.410272 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ae79df-54c86dbff-2sfw2"] Nov 01 16:24:23.604821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.603971 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:23.700300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.700266 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:23.706761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.706716 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77"] Nov 01 16:24:23.711886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.711861 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ae79df-dtr77"] Nov 01 16:24:23.731029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.731001 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:24:23.790554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.790523 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b37c04ce-7216-422c-b973-974e69057427-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37c04ce-7216-422c-b973-974e69057427-openshift-service-ca\") pod \"b37c04ce-7216-422c-b973-974e69057427\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " Nov 01 16:24:23.790733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.790616 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b37c04ce-7216-422c-b973-974e69057427-kube-api-access-5dccl\" (UniqueName: \"kubernetes.io/projected/b37c04ce-7216-422c-b973-974e69057427-kube-api-access-5dccl\") pod \"b37c04ce-7216-422c-b973-974e69057427\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " Nov 01 16:24:23.790733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.790708 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-maas-api-tls\") pod \"b37c04ce-7216-422c-b973-974e69057427\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " Nov 01 16:24:23.790894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.790808 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-metrics-certs\") pod \"b37c04ce-7216-422c-b973-974e69057427\" (UID: \"b37c04ce-7216-422c-b973-974e69057427\") " Nov 01 16:24:23.791028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.791003 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b37c04ce-7216-422c-b973-974e69057427-openshift-service-ca" pod "b37c04ce-7216-422c-b973-974e69057427" (UID: "b37c04ce-7216-422c-b973-974e69057427"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:24:23.791639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.791610 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b37c04ce-7216-422c-b973-974e69057427-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:23.793133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.793104 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-maas-api-tls" pod "b37c04ce-7216-422c-b973-974e69057427" (UID: "b37c04ce-7216-422c-b973-974e69057427"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:24:23.793604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.793566 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b37c04ce-7216-422c-b973-974e69057427-kube-api-access-5dccl" pod "b37c04ce-7216-422c-b973-974e69057427" (UID: "b37c04ce-7216-422c-b973-974e69057427"). InnerVolumeSpecName "kube-api-access-5dccl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:24:23.793702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.793621 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-metrics-certs" pod "b37c04ce-7216-422c-b973-974e69057427" (UID: "b37c04ce-7216-422c-b973-974e69057427"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:24:23.893425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.893388 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:23.893425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.893417 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dccl\" (UniqueName: \"kubernetes.io/projected/b37c04ce-7216-422c-b973-974e69057427-kube-api-access-5dccl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:23.893425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.893431 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b37c04ce-7216-422c-b973-974e69057427-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:24:23.903951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:23.903918 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:24.159855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.159736 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:24.365742 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:24:24.365708 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:24:24.368349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.368321 4573 generic.go:356] "Generic (PLEG): container finished" podID="b37c04ce-7216-422c-b973-974e69057427" containerID="05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c" exitCode=0 Nov 01 16:24:24.368523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.368366 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" Nov 01 16:24:24.368523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.368404 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" event={"ID":"b37c04ce-7216-422c-b973-974e69057427","Type":"ContainerDied","Data":"05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c"} Nov 01 16:24:24.368523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.368458 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5" event={"ID":"b37c04ce-7216-422c-b973-974e69057427","Type":"ContainerDied","Data":"8dc7917858ee4f865bc75194704d2c41dd8af16d8b8625f856e847c1b65249b3"} Nov 01 16:24:24.368523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.368489 4573 scope.go:122] "RemoveContainer" containerID="05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c" Nov 01 16:24:24.381155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.381138 4573 scope.go:122] "RemoveContainer" containerID="05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c" Nov 01 16:24:24.381459 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:24:24.381438 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c\": container with ID starting with 05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c not found: ID does not exist" containerID="05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c" Nov 01 16:24:24.381524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.381468 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c"} err="failed to get container status \"05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c\": rpc error: code = NotFound desc = could not find container \"05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c\": container with ID starting with 05d0b96fdc4d9f4bfd7e355a026027d7e4b61d084055d167eeaca1b893f9df5c not found: ID does not exist" Nov 01 16:24:24.393429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.393408 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5"] Nov 01 16:24:24.401415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.401382 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-ae79df-7c9bd7cfdc-fjsb5"] Nov 01 16:24:24.463279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.463201 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45395229-36ee-4bce-9004-e3be3041e165" path="/var/lib/kubelet/pods/45395229-36ee-4bce-9004-e3be3041e165/volumes" Nov 01 16:24:24.463679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.463666 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b37c04ce-7216-422c-b973-974e69057427" path="/var/lib/kubelet/pods/b37c04ce-7216-422c-b973-974e69057427/volumes" Nov 01 16:24:24.464115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.464101 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2459b92-5db6-4cd7-857e-ae74d138cce2" path="/var/lib/kubelet/pods/d2459b92-5db6-4cd7-857e-ae74d138cce2/volumes" Nov 01 16:24:24.613667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:24.613632 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:25.275174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:25.275140 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:25.927182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:25.927153 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:26.570485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:26.570447 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:27.270553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:27.270475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:27.957127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:27.957093 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:28.571445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:28.571408 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:29.222020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:29.221986 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:29.901779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:29.901713 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:30.360511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:30.360476 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:30.463475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:30.463387 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:30.717041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:30.716905 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:30.819630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:30.816361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:31.561108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:31.561075 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:31.670311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:31.670273 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:32.049608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:32.049558 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:32.147159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:32.147120 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:32.258305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:32.258270 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:32.569571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:32.569540 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:33.279140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:33.279107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:33.966793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:33.966732 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:34.579955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:34.579924 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:35.361823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:35.361786 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:36.063380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:36.063350 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:36.186790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:36.186714 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:36.288743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:36.288711 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:36.904719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:36.904688 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:37.395761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.395684 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:37.439205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.439172 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj"] Nov 01 16:24:37.439932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.439917 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerName="istio-proxy" Nov 01 16:24:37.439988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.439933 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerName="istio-proxy" Nov 01 16:24:37.439988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.439950 4573 cpu_manager.go:414] "containerMap: removing container" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" Nov 01 16:24:37.439988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.439956 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" Nov 01 16:24:37.439988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.439965 4573 cpu_manager.go:414] "containerMap: removing container" podUID="45395229-36ee-4bce-9004-e3be3041e165" containerName="payload-processing" Nov 01 16:24:37.439988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.439971 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45395229-36ee-4bce-9004-e3be3041e165" containerName="payload-processing" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.439995 4573 cpu_manager.go:414] "containerMap: removing container" podUID="88acad1c-61c1-4887-8abc-767ccf45e509" containerName="payload-processing" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440000 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88acad1c-61c1-4887-8abc-767ccf45e509" containerName="payload-processing" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440012 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b37c04ce-7216-422c-b973-974e69057427" containerName="maas-api" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440017 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b37c04ce-7216-422c-b973-974e69057427" containerName="maas-api" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440027 4573 cpu_manager.go:414] "containerMap: removing container" podUID="d2459b92-5db6-4cd7-857e-ae74d138cce2" containerName="revoke-keys" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440032 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2459b92-5db6-4cd7-857e-ae74d138cce2" containerName="revoke-keys" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440132 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fc2fe9-898e-4e4f-92f9-e71133689886" containerName="istio-proxy" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440143 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" Nov 01 16:24:37.440153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440149 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="45395229-36ee-4bce-9004-e3be3041e165" containerName="payload-processing" Nov 01 16:24:37.440449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440155 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" Nov 01 16:24:37.440449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440166 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="88acad1c-61c1-4887-8abc-767ccf45e509" containerName="payload-processing" Nov 01 16:24:37.440449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440172 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b37c04ce-7216-422c-b973-974e69057427" containerName="maas-api" Nov 01 16:24:37.440449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.440179 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2459b92-5db6-4cd7-857e-ae74d138cce2" containerName="revoke-keys" Nov 01 16:24:37.444200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.444176 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:37.454083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.454058 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj"] Nov 01 16:24:37.456182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.455674 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e1e630\"" Nov 01 16:24:37.456425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.456402 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e1e630-dockercfg-ss69m\"" Nov 01 16:24:37.460249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.460228 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm"] Nov 01 16:24:37.460937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.460922 4573 cpu_manager.go:414] "containerMap: removing container" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" Nov 01 16:24:37.460987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.460938 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59186df5-07c5-4dbc-b45d-d474e1514fe6" containerName="istio-proxy" Nov 01 16:24:37.464848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.464831 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:37.474878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.473764 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm"] Nov 01 16:24:37.490136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.490103 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79a8a7f-7e57-4d96-96f9-6904a66e6036-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj\" (UID: \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:37.490528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.490503 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540866f-1428-46c8-86fe-575ad3fb3d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm\" (UID: \"9540866f-1428-46c8-86fe-575ad3fb3d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:37.490651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.490636 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7969\" (UniqueName: \"kubernetes.io/projected/9540866f-1428-46c8-86fe-575ad3fb3d4c-kube-api-access-c7969\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm\" (UID: \"9540866f-1428-46c8-86fe-575ad3fb3d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:37.490812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.490789 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd6nh\" (UniqueName: \"kubernetes.io/projected/c79a8a7f-7e57-4d96-96f9-6904a66e6036-kube-api-access-pd6nh\") pod \"payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj\" (UID: \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:37.492137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.492112 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:37.604770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.602841 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540866f-1428-46c8-86fe-575ad3fb3d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm\" (UID: \"9540866f-1428-46c8-86fe-575ad3fb3d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:37.604770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.602955 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7969\" (UniqueName: \"kubernetes.io/projected/9540866f-1428-46c8-86fe-575ad3fb3d4c-kube-api-access-c7969\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm\" (UID: \"9540866f-1428-46c8-86fe-575ad3fb3d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:37.604770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.603041 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd6nh\" (UniqueName: \"kubernetes.io/projected/c79a8a7f-7e57-4d96-96f9-6904a66e6036-kube-api-access-pd6nh\") pod \"payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj\" (UID: \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:37.604770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.603167 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79a8a7f-7e57-4d96-96f9-6904a66e6036-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj\" (UID: \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:37.604770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.604128 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79a8a7f-7e57-4d96-96f9-6904a66e6036-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj\" (UID: \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:37.628783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.605332 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540866f-1428-46c8-86fe-575ad3fb3d4c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm\" (UID: \"9540866f-1428-46c8-86fe-575ad3fb3d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:37.655823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.636489 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7969\" (UniqueName: \"kubernetes.io/projected/9540866f-1428-46c8-86fe-575ad3fb3d4c-kube-api-access-c7969\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm\" (UID: \"9540866f-1428-46c8-86fe-575ad3fb3d4c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:37.684773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.678946 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd6nh\" (UniqueName: \"kubernetes.io/projected/c79a8a7f-7e57-4d96-96f9-6904a66e6036-kube-api-access-pd6nh\") pod \"payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj\" (UID: \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:37.763773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.762892 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:37.793257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.793230 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:37.989996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:37.989945 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:38.105251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.100841 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj"] Nov 01 16:24:38.136314 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:24:38.136281 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9540866f_1428_46c8_86fe_575ad3fb3d4c.slice/crio-78635305f647bd6101dff5085f7074689e0b8d3991a42040ee6ed05683e4c1a6 WatchSource:0}: Error finding container 78635305f647bd6101dff5085f7074689e0b8d3991a42040ee6ed05683e4c1a6: Status 404 returned error can't find the container with id 78635305f647bd6101dff5085f7074689e0b8d3991a42040ee6ed05683e4c1a6 Nov 01 16:24:38.137077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.137051 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm"] Nov 01 16:24:38.533390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.533281 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" event={"ID":"c79a8a7f-7e57-4d96-96f9-6904a66e6036","Type":"ContainerStarted","Data":"72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998"} Nov 01 16:24:38.533390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.533349 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" event={"ID":"c79a8a7f-7e57-4d96-96f9-6904a66e6036","Type":"ContainerStarted","Data":"783ac079e5e2f2a613c135dfbb60cdee686e434c10c820fdfe16ac719c61e790"} Nov 01 16:24:38.533390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.533378 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:38.535080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.535049 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" event={"ID":"9540866f-1428-46c8-86fe-575ad3fb3d4c","Type":"ContainerStarted","Data":"3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667"} Nov 01 16:24:38.535223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.535098 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" event={"ID":"9540866f-1428-46c8-86fe-575ad3fb3d4c","Type":"ContainerStarted","Data":"78635305f647bd6101dff5085f7074689e0b8d3991a42040ee6ed05683e4c1a6"} Nov 01 16:24:38.535223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.535137 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:38.551692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.551649 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" podStartSLOduration=1.551636719 podStartE2EDuration="1.551636719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:24:38.550606149 +0000 UTC m=+2216.253772087" watchObservedRunningTime="2026-11-01 16:24:38.551636719 +0000 UTC m=+2216.254802664" Nov 01 16:24:38.568666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.568607 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" podStartSLOduration=1.568594153 podStartE2EDuration="1.568594153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:24:38.565107286 +0000 UTC m=+2216.268273233" watchObservedRunningTime="2026-11-01 16:24:38.568594153 +0000 UTC m=+2216.271760136" Nov 01 16:24:38.803637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:38.803544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:39.160744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:39.160713 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:39.261248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:39.261215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:39.457558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:39.457448 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:40.208782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:40.208737 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:40.961385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:40.961349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:41.635074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:41.635038 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:42.297402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:42.297335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:43.269495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:43.269457 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:44.110773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:44.098054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:44.975452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:44.975407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:45.959009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:45.958964 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:46.700142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:46.700107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:47.449740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:47.449690 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:48.159897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.159860 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:48.193416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.193389 4573 scope.go:122] "RemoveContainer" containerID="beb2cf313cc193cd6de95313650df2dbe88e157e699052bf931cc1c6a84ed4df" Nov 01 16:24:48.205199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.205181 4573 scope.go:122] "RemoveContainer" containerID="9035c112329e8abd0f67ca5849bd46c90d6c13516e035ba91aabc733dd755328" Nov 01 16:24:48.227072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.218090 4573 scope.go:122] "RemoveContainer" containerID="d1e6bfee60d4f2f280913d910a33561f3f5fff9b0c2bd33de371835dfd63cc7d" Nov 01 16:24:48.273973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.272919 4573 scope.go:122] "RemoveContainer" containerID="8751bba9f070f31f958a9da22036142d02307a22cfaec9abcbadf213807bd830" Nov 01 16:24:48.290511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.290483 4573 scope.go:122] "RemoveContainer" containerID="0370bb85c298779f85078735d8bd8dd6cdf970d0f6f36b1632a527daefca4ef2" Nov 01 16:24:48.300841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.300819 4573 scope.go:122] "RemoveContainer" containerID="691093e8b1615079c77ecbacbe72a3ba7cfd235027666cee974c386ab1fb31d2" Nov 01 16:24:48.310491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.310472 4573 scope.go:122] "RemoveContainer" containerID="53df8e7c9ef6a3a84f7093c19069098f7362dd9b1478c53f669fe9e284d97287" Nov 01 16:24:48.778558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:48.778504 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:49.453154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:49.453090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:49.572641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:49.572599 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:24:49.572858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:49.572680 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:24:50.172812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.172780 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:50.822828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.822792 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv"] Nov 01 16:24:50.841021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.840993 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv"] Nov 01 16:24:50.841407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.841343 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:50.847593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.847573 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e1e630-serving-cert\"" Nov 01 16:24:50.868277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.868245 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:50.909670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.909634 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719e9e9e-90ae-4178-b113-96166fdeb425-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:50.909804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.909776 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpftt\" (UniqueName: \"kubernetes.io/projected/719e9e9e-90ae-4178-b113-96166fdeb425-kube-api-access-kpftt\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:50.909913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.909898 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:50.909953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.909937 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:50.943284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:50.943256 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:51.010814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.010788 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.011000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.010976 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719e9e9e-90ae-4178-b113-96166fdeb425-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.011080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.011064 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpftt\" (UniqueName: \"kubernetes.io/projected/719e9e9e-90ae-4178-b113-96166fdeb425-kube-api-access-kpftt\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.011188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.011172 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.011963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.011937 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719e9e9e-90ae-4178-b113-96166fdeb425-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.014297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.014272 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.031884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.030313 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-metrics-certs\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.032086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.031990 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpftt\" (UniqueName: \"kubernetes.io/projected/719e9e9e-90ae-4178-b113-96166fdeb425-kube-api-access-kpftt\") pod \"maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.049086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.048947 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:51.161021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.160988 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:51.465957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.464546 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv"] Nov 01 16:24:51.687568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.687535 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:51.687804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:51.687649 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" event={"ID":"719e9e9e-90ae-4178-b113-96166fdeb425","Type":"ContainerStarted","Data":"6d799361a162ff9aae326e95290f925022c5a1fa7752841c53ef1c0e9463c448"} Nov 01 16:24:52.434956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:52.434918 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:52.698468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:52.698378 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" event={"ID":"719e9e9e-90ae-4178-b113-96166fdeb425","Type":"ContainerStarted","Data":"9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5"} Nov 01 16:24:52.698468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:52.698449 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:52.718929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:52.718880 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" podStartSLOduration=2.247828825 podStartE2EDuration="2.71886708s" totalImagesPullingTime="471.038255ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:24:52.714401299 +0000 UTC m=+2230.417567250" watchObservedRunningTime="2026-11-01 16:24:52.71886708 +0000 UTC m=+2230.422033015" Nov 01 16:24:53.144960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:53.144916 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:53.765387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:53.765352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:54.364819 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:24:54.364784 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:24:54.546902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:54.546839 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:55.343705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:55.343667 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:56.123457 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:56.123429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:56.883516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:56.883486 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:57.666850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:57.666745 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:58.306474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:58.306424 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:58.713147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:58.713117 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:24:58.987259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:58.987172 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:24:59.666228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:24:59.666194 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:00.281408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:00.281370 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:00.964922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:00.964852 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:01.675274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:01.675238 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:02.362184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:02.362137 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:03.152894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:03.152845 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:03.926621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:03.926588 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:04.660173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:04.660137 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:05.458688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:05.458623 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:06.139869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:06.139817 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:06.899563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:06.899527 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:07.762665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:07.762629 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:08.415445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:08.415411 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:09.162554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:09.162521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:09.880668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:09.880630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:10.540409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:10.540376 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:11.181989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:11.181948 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:11.558844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:11.558738 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj"] Nov 01 16:25:11.582031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:11.578655 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm"] Nov 01 16:25:11.618530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:11.618484 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:11.717538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:11.717505 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:11.884771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:11.884724 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:11.981456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:11.978787 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:12.840694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:12.840655 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:13.471929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:13.471896 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:14.030573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:14.030534 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf"] Nov 01 16:25:14.034633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:14.034611 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09fdf186-7lfpf"] Nov 01 16:25:14.177499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:14.177458 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:14.460258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:14.460225 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5cd9e54-98f6-42f0-8ad3-af7939680840" path="/var/lib/kubelet/pods/a5cd9e54-98f6-42f0-8ad3-af7939680840/volumes" Nov 01 16:25:14.861921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:14.861888 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:15.543678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:15.543649 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:16.264946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.264914 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:16.559896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.559743 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" podUID="c79a8a7f-7e57-4d96-96f9-6904a66e6036" containerName="payload-processing" containerID="cri-o://72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998" gracePeriod=25 Nov 01 16:25:16.579552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.579517 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" podUID="9540866f-1428-46c8-86fe-575ad3fb3d4c" containerName="payload-pre-processing" containerID="cri-o://3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667" gracePeriod=25 Nov 01 16:25:16.871463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.871429 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:25:16.912949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.912920 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:25:16.984892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.984859 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:16.995203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.995178 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79a8a7f-7e57-4d96-96f9-6904a66e6036-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79a8a7f-7e57-4d96-96f9-6904a66e6036-plugins-config-volume\") pod \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\" (UID: \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\") " Nov 01 16:25:16.995363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.995252 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9540866f-1428-46c8-86fe-575ad3fb3d4c-kube-api-access-c7969\" (UniqueName: \"kubernetes.io/projected/9540866f-1428-46c8-86fe-575ad3fb3d4c-kube-api-access-c7969\") pod \"9540866f-1428-46c8-86fe-575ad3fb3d4c\" (UID: \"9540866f-1428-46c8-86fe-575ad3fb3d4c\") " Nov 01 16:25:16.995429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.995361 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9540866f-1428-46c8-86fe-575ad3fb3d4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540866f-1428-46c8-86fe-575ad3fb3d4c-plugins-config-volume\") pod \"9540866f-1428-46c8-86fe-575ad3fb3d4c\" (UID: \"9540866f-1428-46c8-86fe-575ad3fb3d4c\") " Nov 01 16:25:16.995429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.995400 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79a8a7f-7e57-4d96-96f9-6904a66e6036-kube-api-access-pd6nh\" (UniqueName: \"kubernetes.io/projected/c79a8a7f-7e57-4d96-96f9-6904a66e6036-kube-api-access-pd6nh\") pod \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\" (UID: \"c79a8a7f-7e57-4d96-96f9-6904a66e6036\") " Nov 01 16:25:16.995629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.995599 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79a8a7f-7e57-4d96-96f9-6904a66e6036-plugins-config-volume" pod "c79a8a7f-7e57-4d96-96f9-6904a66e6036" (UID: "c79a8a7f-7e57-4d96-96f9-6904a66e6036"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:25:16.995793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.995738 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9540866f-1428-46c8-86fe-575ad3fb3d4c-plugins-config-volume" pod "9540866f-1428-46c8-86fe-575ad3fb3d4c" (UID: "9540866f-1428-46c8-86fe-575ad3fb3d4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:25:16.996175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.996151 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c79a8a7f-7e57-4d96-96f9-6904a66e6036-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:16.996280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.996176 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9540866f-1428-46c8-86fe-575ad3fb3d4c-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:16.997615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.997595 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9540866f-1428-46c8-86fe-575ad3fb3d4c-kube-api-access-c7969" pod "9540866f-1428-46c8-86fe-575ad3fb3d4c" (UID: "9540866f-1428-46c8-86fe-575ad3fb3d4c"). InnerVolumeSpecName "kube-api-access-c7969". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:25:16.997709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:16.997623 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79a8a7f-7e57-4d96-96f9-6904a66e6036-kube-api-access-pd6nh" pod "c79a8a7f-7e57-4d96-96f9-6904a66e6036" (UID: "c79a8a7f-7e57-4d96-96f9-6904a66e6036"). InnerVolumeSpecName "kube-api-access-pd6nh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:25:17.004626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.004599 4573 generic.go:356] "Generic (PLEG): container finished" podID="c79a8a7f-7e57-4d96-96f9-6904a66e6036" containerID="72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998" exitCode=0 Nov 01 16:25:17.004732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.004652 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" Nov 01 16:25:17.004732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.004682 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" event={"ID":"c79a8a7f-7e57-4d96-96f9-6904a66e6036","Type":"ContainerDied","Data":"72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998"} Nov 01 16:25:17.004894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.004783 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj" event={"ID":"c79a8a7f-7e57-4d96-96f9-6904a66e6036","Type":"ContainerDied","Data":"783ac079e5e2f2a613c135dfbb60cdee686e434c10c820fdfe16ac719c61e790"} Nov 01 16:25:17.004894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.004813 4573 scope.go:122] "RemoveContainer" containerID="72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998" Nov 01 16:25:17.006522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.006499 4573 generic.go:356] "Generic (PLEG): container finished" podID="9540866f-1428-46c8-86fe-575ad3fb3d4c" containerID="3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667" exitCode=0 Nov 01 16:25:17.006627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.006550 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" Nov 01 16:25:17.006627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.006594 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" event={"ID":"9540866f-1428-46c8-86fe-575ad3fb3d4c","Type":"ContainerDied","Data":"3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667"} Nov 01 16:25:17.006741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.006658 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm" event={"ID":"9540866f-1428-46c8-86fe-575ad3fb3d4c","Type":"ContainerDied","Data":"78635305f647bd6101dff5085f7074689e0b8d3991a42040ee6ed05683e4c1a6"} Nov 01 16:25:17.018652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.018638 4573 scope.go:122] "RemoveContainer" containerID="72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998" Nov 01 16:25:17.018941 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:25:17.018923 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998\": container with ID starting with 72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998 not found: ID does not exist" containerID="72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998" Nov 01 16:25:17.019006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.018948 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998"} err="failed to get container status \"72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998\": rpc error: code = NotFound desc = could not find container \"72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998\": container with ID starting with 72a7e9ed9b7504babfe81791cfcd9ebe699b69b17798cf87806e517958336998 not found: ID does not exist" Nov 01 16:25:17.019006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.018962 4573 scope.go:122] "RemoveContainer" containerID="3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667" Nov 01 16:25:17.029688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.029672 4573 scope.go:122] "RemoveContainer" containerID="3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667" Nov 01 16:25:17.029959 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:25:17.029938 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667\": container with ID starting with 3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667 not found: ID does not exist" containerID="3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667" Nov 01 16:25:17.030042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.029975 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667"} err="failed to get container status \"3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667\": rpc error: code = NotFound desc = could not find container \"3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667\": container with ID starting with 3b8241536b4081ffb25960214eb403c3e87b046b35996a073327b1e8d31ac667 not found: ID does not exist" Nov 01 16:25:17.032869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.032849 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj"] Nov 01 16:25:17.075473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.074824 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-7d69467b6f-fq7jj"] Nov 01 16:25:17.092126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.092095 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm"] Nov 01 16:25:17.097021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.096989 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-665899d59-cmmgm"] Nov 01 16:25:17.097560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.097533 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7969\" (UniqueName: \"kubernetes.io/projected/9540866f-1428-46c8-86fe-575ad3fb3d4c-kube-api-access-c7969\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:17.097673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.097565 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd6nh\" (UniqueName: \"kubernetes.io/projected/c79a8a7f-7e57-4d96-96f9-6904a66e6036-kube-api-access-pd6nh\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:17.427785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.424577 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:25:17.875452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:17.875421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:18.460514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:18.460479 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9540866f-1428-46c8-86fe-575ad3fb3d4c" path="/var/lib/kubelet/pods/9540866f-1428-46c8-86fe-575ad3fb3d4c/volumes" Nov 01 16:25:18.460952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:18.460937 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79a8a7f-7e57-4d96-96f9-6904a66e6036" path="/var/lib/kubelet/pods/c79a8a7f-7e57-4d96-96f9-6904a66e6036/volumes" Nov 01 16:25:19.068236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:19.068200 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:19.977027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:19.976995 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:20.849806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:20.849772 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:21.562037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:21.562006 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:22.241933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.241896 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:22.834475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.834205 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:22.927371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.927336 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:22.931604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.931577 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm"] Nov 01 16:25:22.932354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.932338 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c79a8a7f-7e57-4d96-96f9-6904a66e6036" containerName="payload-processing" Nov 01 16:25:22.932408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.932355 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79a8a7f-7e57-4d96-96f9-6904a66e6036" containerName="payload-processing" Nov 01 16:25:22.932408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.932363 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9540866f-1428-46c8-86fe-575ad3fb3d4c" containerName="payload-pre-processing" Nov 01 16:25:22.932408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.932368 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9540866f-1428-46c8-86fe-575ad3fb3d4c" containerName="payload-pre-processing" Nov 01 16:25:22.932512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.932502 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79a8a7f-7e57-4d96-96f9-6904a66e6036" containerName="payload-processing" Nov 01 16:25:22.932551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.932513 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9540866f-1428-46c8-86fe-575ad3fb3d4c" containerName="payload-pre-processing" Nov 01 16:25:22.937836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.937819 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:22.943652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.943629 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm"] Nov 01 16:25:22.946157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.946138 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e1e630\"" Nov 01 16:25:22.959131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.959109 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm"] Nov 01 16:25:22.966975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.966954 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:22.976287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.976264 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e1e630-dockercfg-vk2t2\"" Nov 01 16:25:22.983829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:22.983180 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm"] Nov 01 16:25:23.026057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.026023 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df545125-bbe8-4ed1-8d81-012278b7eeaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm\" (UID: \"df545125-bbe8-4ed1-8d81-012278b7eeaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:23.026226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.026180 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kc89\" (UniqueName: \"kubernetes.io/projected/6768f9ea-68d1-4f33-8e92-fa224878ee1c-kube-api-access-5kc89\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm\" (UID: \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:23.026335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.026309 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjcf2\" (UniqueName: \"kubernetes.io/projected/df545125-bbe8-4ed1-8d81-012278b7eeaa-kube-api-access-bjcf2\") pod \"payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm\" (UID: \"df545125-bbe8-4ed1-8d81-012278b7eeaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:23.026461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.026443 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768f9ea-68d1-4f33-8e92-fa224878ee1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm\" (UID: \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:23.151776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.147850 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjcf2\" (UniqueName: \"kubernetes.io/projected/df545125-bbe8-4ed1-8d81-012278b7eeaa-kube-api-access-bjcf2\") pod \"payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm\" (UID: \"df545125-bbe8-4ed1-8d81-012278b7eeaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:23.151776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.147980 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768f9ea-68d1-4f33-8e92-fa224878ee1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm\" (UID: \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:23.151776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.148190 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df545125-bbe8-4ed1-8d81-012278b7eeaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm\" (UID: \"df545125-bbe8-4ed1-8d81-012278b7eeaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:23.151776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.148274 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kc89\" (UniqueName: \"kubernetes.io/projected/6768f9ea-68d1-4f33-8e92-fa224878ee1c-kube-api-access-5kc89\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm\" (UID: \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:23.171552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.167330 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df545125-bbe8-4ed1-8d81-012278b7eeaa-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm\" (UID: \"df545125-bbe8-4ed1-8d81-012278b7eeaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:23.172406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.172378 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768f9ea-68d1-4f33-8e92-fa224878ee1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm\" (UID: \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:23.188591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.184032 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kc89\" (UniqueName: \"kubernetes.io/projected/6768f9ea-68d1-4f33-8e92-fa224878ee1c-kube-api-access-5kc89\") pod \"payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm\" (UID: \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:23.217407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.217371 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjcf2\" (UniqueName: \"kubernetes.io/projected/df545125-bbe8-4ed1-8d81-012278b7eeaa-kube-api-access-bjcf2\") pod \"payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm\" (UID: \"df545125-bbe8-4ed1-8d81-012278b7eeaa\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:23.265775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.259969 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:23.283670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.283642 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:23.483631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.483596 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:23.598831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.598803 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm"] Nov 01 16:25:23.599374 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:25:23.599345 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6768f9ea_68d1_4f33_8e92_fa224878ee1c.slice/crio-627dded5836229bf59c1a23210d5f97286f26797fd9bab7d43884127084d69c7 WatchSource:0}: Error finding container 627dded5836229bf59c1a23210d5f97286f26797fd9bab7d43884127084d69c7: Status 404 returned error can't find the container with id 627dded5836229bf59c1a23210d5f97286f26797fd9bab7d43884127084d69c7 Nov 01 16:25:23.600610 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:25:23.600566 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf545125_bbe8_4ed1_8d81_012278b7eeaa.slice/crio-7e012681f57121f1c3b6584be866ac4d49c1fb253aa4e85d171f78f1cfc654d2 WatchSource:0}: Error finding container 7e012681f57121f1c3b6584be866ac4d49c1fb253aa4e85d171f78f1cfc654d2: Status 404 returned error can't find the container with id 7e012681f57121f1c3b6584be866ac4d49c1fb253aa4e85d171f78f1cfc654d2 Nov 01 16:25:23.604867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.604847 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm"] Nov 01 16:25:23.874033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.874003 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q"] Nov 01 16:25:23.880171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.880144 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:23.885883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.884161 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:25:23.885883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.884199 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:25:23.886562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.886535 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q"] Nov 01 16:25:23.973370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.973339 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:23.973588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.973412 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:23.973672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:23.973603 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxt6v\" (UniqueName: \"kubernetes.io/projected/ea0c76cc-4a3b-4205-99ff-850da03881b6-kube-api-access-cxt6v\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:24.074878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.074843 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:24.075104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.074912 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:24.075104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.074994 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxt6v\" (UniqueName: \"kubernetes.io/projected/ea0c76cc-4a3b-4205-99ff-850da03881b6-kube-api-access-cxt6v\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:24.075673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.075648 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:24.077340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.077316 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:24.083250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.083212 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxt6v\" (UniqueName: \"kubernetes.io/projected/ea0c76cc-4a3b-4205-99ff-850da03881b6-kube-api-access-cxt6v\") pod \"maas-api-revoke-sub-ce8f835d-dlb4q\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:24.177660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.177480 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" event={"ID":"6768f9ea-68d1-4f33-8e92-fa224878ee1c","Type":"ContainerStarted","Data":"627dded5836229bf59c1a23210d5f97286f26797fd9bab7d43884127084d69c7"} Nov 01 16:25:24.179421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.179387 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" event={"ID":"df545125-bbe8-4ed1-8d81-012278b7eeaa","Type":"ContainerStarted","Data":"7e012681f57121f1c3b6584be866ac4d49c1fb253aa4e85d171f78f1cfc654d2"} Nov 01 16:25:24.189832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.189780 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:24.202881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.202855 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:24.391336 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:25:24.391300 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:25:24.576686 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:25:24.576649 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0c76cc_4a3b_4205_99ff_850da03881b6.slice/crio-fa316b2fc708a9081296493147dd9d7b0ab8064ff4a779ccbeba8055b4a12b21 WatchSource:0}: Error finding container fa316b2fc708a9081296493147dd9d7b0ab8064ff4a779ccbeba8055b4a12b21: Status 404 returned error can't find the container with id fa316b2fc708a9081296493147dd9d7b0ab8064ff4a779ccbeba8055b4a12b21 Nov 01 16:25:24.578539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.578515 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q"] Nov 01 16:25:24.852818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:24.852711 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:25.197930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:25.197887 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" event={"ID":"ea0c76cc-4a3b-4205-99ff-850da03881b6","Type":"ContainerStarted","Data":"5f88c347dc06fcb05bb509c6271219d9dd8e1f0a046ad11fd844b4865492b459"} Nov 01 16:25:25.198146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:25.197959 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" event={"ID":"ea0c76cc-4a3b-4205-99ff-850da03881b6","Type":"ContainerStarted","Data":"fa316b2fc708a9081296493147dd9d7b0ab8064ff4a779ccbeba8055b4a12b21"} Nov 01 16:25:25.221538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:25.221481 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" podStartSLOduration=2.22146243 podStartE2EDuration="2.22146243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:25.2150443 +0000 UTC m=+2262.918210238" watchObservedRunningTime="2026-11-01 16:25:25.22146243 +0000 UTC m=+2262.924628368" Nov 01 16:25:25.579209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:25.579109 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:26.211588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.211537 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" event={"ID":"6768f9ea-68d1-4f33-8e92-fa224878ee1c","Type":"ContainerStarted","Data":"5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8"} Nov 01 16:25:26.211794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.211617 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:26.213679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.213654 4573 generic.go:356] "Generic (PLEG): container finished" podID="ea0c76cc-4a3b-4205-99ff-850da03881b6" containerID="5f88c347dc06fcb05bb509c6271219d9dd8e1f0a046ad11fd844b4865492b459" exitCode=0 Nov 01 16:25:26.213831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.213740 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" event={"ID":"ea0c76cc-4a3b-4205-99ff-850da03881b6","Type":"ContainerDied","Data":"5f88c347dc06fcb05bb509c6271219d9dd8e1f0a046ad11fd844b4865492b459"} Nov 01 16:25:26.215647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.215621 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" event={"ID":"df545125-bbe8-4ed1-8d81-012278b7eeaa","Type":"ContainerStarted","Data":"d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b"} Nov 01 16:25:26.215796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.215712 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:26.229214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.229166 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" podStartSLOduration=1.80517897 podStartE2EDuration="4.229152744s" totalImagesPullingTime="2.423973774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:26.22781998 +0000 UTC m=+2263.930985920" watchObservedRunningTime="2026-11-01 16:25:26.229152744 +0000 UTC m=+2263.932318679" Nov 01 16:25:26.245085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.245032 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" podStartSLOduration=1.78156067 podStartE2EDuration="4.245014268s" totalImagesPullingTime="2.463453598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:26.243200031 +0000 UTC m=+2263.946365968" watchObservedRunningTime="2026-11-01 16:25:26.245014268 +0000 UTC m=+2263.948180205" Nov 01 16:25:26.351431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:26.351355 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:27.017829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.017789 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:27.419802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.419768 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:27.607096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.607069 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-xks-ca\") pod \"ea0c76cc-4a3b-4205-99ff-850da03881b6\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " Nov 01 16:25:27.607296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.607261 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0c76cc-4a3b-4205-99ff-850da03881b6-kube-api-access-cxt6v\" (UniqueName: \"kubernetes.io/projected/ea0c76cc-4a3b-4205-99ff-850da03881b6-kube-api-access-cxt6v\") pod \"ea0c76cc-4a3b-4205-99ff-850da03881b6\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " Nov 01 16:25:27.607371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.607358 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-openshift-ca\") pod \"ea0c76cc-4a3b-4205-99ff-850da03881b6\" (UID: \"ea0c76cc-4a3b-4205-99ff-850da03881b6\") " Nov 01 16:25:27.607721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.607697 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-openshift-ca" pod "ea0c76cc-4a3b-4205-99ff-850da03881b6" (UID: "ea0c76cc-4a3b-4205-99ff-850da03881b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:25:27.608092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.608067 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:27.609844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.609818 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-xks-ca" pod "ea0c76cc-4a3b-4205-99ff-850da03881b6" (UID: "ea0c76cc-4a3b-4205-99ff-850da03881b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:25:27.610014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.609895 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0c76cc-4a3b-4205-99ff-850da03881b6-kube-api-access-cxt6v" pod "ea0c76cc-4a3b-4205-99ff-850da03881b6" (UID: "ea0c76cc-4a3b-4205-99ff-850da03881b6"). InnerVolumeSpecName "kube-api-access-cxt6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:25:27.685563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.685533 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:27.709613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.709582 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxt6v\" (UniqueName: \"kubernetes.io/projected/ea0c76cc-4a3b-4205-99ff-850da03881b6-kube-api-access-cxt6v\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:27.709613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:27.709616 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea0c76cc-4a3b-4205-99ff-850da03881b6-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:28.238036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:28.238005 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" Nov 01 16:25:28.238036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:28.238021 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q" event={"ID":"ea0c76cc-4a3b-4205-99ff-850da03881b6","Type":"ContainerDied","Data":"fa316b2fc708a9081296493147dd9d7b0ab8064ff4a779ccbeba8055b4a12b21"} Nov 01 16:25:28.238531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:28.238080 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa316b2fc708a9081296493147dd9d7b0ab8064ff4a779ccbeba8055b4a12b21" Nov 01 16:25:28.420598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:28.420551 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:29.061422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:29.061385 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:29.942626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:29.942567 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:30.575276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:30.575234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:31.282347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:31.282313 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:32.072445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:32.072411 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:32.177069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:32.177040 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:32.711465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:32.711430 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:33.023922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:33.023846 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5"] Nov 01 16:25:33.028032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:33.028006 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce829a37-89nb5"] Nov 01 16:25:33.394413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:33.394378 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:34.148144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.148108 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:34.157593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.157566 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p"] Nov 01 16:25:34.158427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.158410 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ea0c76cc-4a3b-4205-99ff-850da03881b6" containerName="revoke-keys" Nov 01 16:25:34.158501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.158429 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0c76cc-4a3b-4205-99ff-850da03881b6" containerName="revoke-keys" Nov 01 16:25:34.158544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.158536 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0c76cc-4a3b-4205-99ff-850da03881b6" containerName="revoke-keys" Nov 01 16:25:34.162610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.162592 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.166983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.166962 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:25:34.167130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.167048 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:25:34.170325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.170299 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p"] Nov 01 16:25:34.349044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.349013 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9nhk\" (UniqueName: \"kubernetes.io/projected/5b143285-fc25-4acb-aecc-22c0fe812f3c-kube-api-access-p9nhk\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.349285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.349258 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.349361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.349312 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.451191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.451112 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9nhk\" (UniqueName: \"kubernetes.io/projected/5b143285-fc25-4acb-aecc-22c0fe812f3c-kube-api-access-p9nhk\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.451391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.451257 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.451391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.451287 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.452151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.452124 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.453681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.453651 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.459394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.459374 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9nhk\" (UniqueName: \"kubernetes.io/projected/5b143285-fc25-4acb-aecc-22c0fe812f3c-kube-api-access-p9nhk\") pod \"maas-api-revoke-sub-1754432c-7w57p\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.461898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.461866 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69251bb9-549e-4499-913a-1ab14c566987" path="/var/lib/kubelet/pods/69251bb9-549e-4499-913a-1ab14c566987/volumes" Nov 01 16:25:34.480917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.480894 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:34.769986 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:25:34.769940 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b143285_fc25_4acb_aecc_22c0fe812f3c.slice/crio-f3670a6c941a190689e78b11bffdb383e1479ef0675902945a4c80b43fd17273 WatchSource:0}: Error finding container f3670a6c941a190689e78b11bffdb383e1479ef0675902945a4c80b43fd17273: Status 404 returned error can't find the container with id f3670a6c941a190689e78b11bffdb383e1479ef0675902945a4c80b43fd17273 Nov 01 16:25:34.770133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.770116 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p"] Nov 01 16:25:34.869340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:34.869309 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:35.301824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:35.301785 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" event={"ID":"5b143285-fc25-4acb-aecc-22c0fe812f3c","Type":"ContainerStarted","Data":"07473e4cdbff615c5a4e13021d574d379371cd1d7d1ac38ba33c4e49b2101cd2"} Nov 01 16:25:35.302345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:35.301855 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" event={"ID":"5b143285-fc25-4acb-aecc-22c0fe812f3c","Type":"ContainerStarted","Data":"f3670a6c941a190689e78b11bffdb383e1479ef0675902945a4c80b43fd17273"} Nov 01 16:25:35.334407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:35.334336 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" podStartSLOduration=1.33431723 podStartE2EDuration="1.33431723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:35.327563637 +0000 UTC m=+2273.030729574" watchObservedRunningTime="2026-11-01 16:25:35.33431723 +0000 UTC m=+2273.037483170" Nov 01 16:25:35.590431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:35.590346 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:36.279133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:36.279095 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:36.312428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:36.312397 4573 generic.go:356] "Generic (PLEG): container finished" podID="5b143285-fc25-4acb-aecc-22c0fe812f3c" containerID="07473e4cdbff615c5a4e13021d574d379371cd1d7d1ac38ba33c4e49b2101cd2" exitCode=0 Nov 01 16:25:36.312905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:36.312425 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" event={"ID":"5b143285-fc25-4acb-aecc-22c0fe812f3c","Type":"ContainerDied","Data":"07473e4cdbff615c5a4e13021d574d379371cd1d7d1ac38ba33c4e49b2101cd2"} Nov 01 16:25:36.966368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:36.966330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:37.227903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.227809 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:25:37.227903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.227896 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:25:37.498885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.498823 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:37.613072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.613040 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b143285-fc25-4acb-aecc-22c0fe812f3c-kube-api-access-p9nhk\" (UniqueName: \"kubernetes.io/projected/5b143285-fc25-4acb-aecc-22c0fe812f3c-kube-api-access-p9nhk\") pod \"5b143285-fc25-4acb-aecc-22c0fe812f3c\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " Nov 01 16:25:37.613272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.613259 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-xks-ca\") pod \"5b143285-fc25-4acb-aecc-22c0fe812f3c\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " Nov 01 16:25:37.613367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.613352 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-openshift-ca\") pod \"5b143285-fc25-4acb-aecc-22c0fe812f3c\" (UID: \"5b143285-fc25-4acb-aecc-22c0fe812f3c\") " Nov 01 16:25:37.613780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.613735 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-openshift-ca" pod "5b143285-fc25-4acb-aecc-22c0fe812f3c" (UID: "5b143285-fc25-4acb-aecc-22c0fe812f3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:25:37.614099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.614080 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:37.615339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.615312 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-xks-ca" pod "5b143285-fc25-4acb-aecc-22c0fe812f3c" (UID: "5b143285-fc25-4acb-aecc-22c0fe812f3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:25:37.615513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.615490 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b143285-fc25-4acb-aecc-22c0fe812f3c-kube-api-access-p9nhk" pod "5b143285-fc25-4acb-aecc-22c0fe812f3c" (UID: "5b143285-fc25-4acb-aecc-22c0fe812f3c"). InnerVolumeSpecName "kube-api-access-p9nhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:25:37.659552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.659521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:37.716123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.716090 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9nhk\" (UniqueName: \"kubernetes.io/projected/5b143285-fc25-4acb-aecc-22c0fe812f3c-kube-api-access-p9nhk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:37.716352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:37.716334 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b143285-fc25-4acb-aecc-22c0fe812f3c-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:25:38.334011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:38.333978 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" event={"ID":"5b143285-fc25-4acb-aecc-22c0fe812f3c","Type":"ContainerDied","Data":"f3670a6c941a190689e78b11bffdb383e1479ef0675902945a4c80b43fd17273"} Nov 01 16:25:38.334011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:38.334008 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p" Nov 01 16:25:38.334232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:38.334028 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3670a6c941a190689e78b11bffdb383e1479ef0675902945a4c80b43fd17273" Nov 01 16:25:38.493118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:38.493084 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:38.595697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:38.595612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:38.910637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:38.910604 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:39.045154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:39.045120 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:39.507305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:39.507274 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:40.246899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:40.246859 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:40.957521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:40.957468 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:41.879423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:41.879393 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:42.568304 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:42.568274 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:43.118387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:43.118352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:43.217403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:43.217370 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:43.565885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:43.565804 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:44.299256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.299225 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:44.380244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.380207 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg"] Nov 01 16:25:44.381111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.381087 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5b143285-fc25-4acb-aecc-22c0fe812f3c" containerName="revoke-keys" Nov 01 16:25:44.381111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.381108 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b143285-fc25-4acb-aecc-22c0fe812f3c" containerName="revoke-keys" Nov 01 16:25:44.381276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.381247 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b143285-fc25-4acb-aecc-22c0fe812f3c" containerName="revoke-keys" Nov 01 16:25:44.385353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.385332 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:44.391999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.390180 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b94b5-dockercfg-fxtlj\"" Nov 01 16:25:44.392906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.392884 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2b94b5\"" Nov 01 16:25:44.396526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.396497 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg"] Nov 01 16:25:44.401590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.401562 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:44.417948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.417908 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps"] Nov 01 16:25:44.446093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.445194 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:44.449174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.449144 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps"] Nov 01 16:25:44.637289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.637248 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg\" (UID: \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:44.637466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.637314 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps\" (UID: \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:44.637466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.637394 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25kch\" (UniqueName: \"kubernetes.io/projected/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-kube-api-access-25kch\") pod \"payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps\" (UID: \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:44.637554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.637470 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctzg\" (UniqueName: \"kubernetes.io/projected/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-kube-api-access-vctzg\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg\" (UID: \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:44.738843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.738806 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg\" (UID: \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:44.738843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.738855 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps\" (UID: \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:44.739115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.738900 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25kch\" (UniqueName: \"kubernetes.io/projected/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-kube-api-access-25kch\") pod \"payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps\" (UID: \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:44.739115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.738969 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctzg\" (UniqueName: \"kubernetes.io/projected/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-kube-api-access-vctzg\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg\" (UID: \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:44.739920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.739885 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps\" (UID: \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:44.739920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.739899 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg\" (UID: \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:44.747446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.747418 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctzg\" (UniqueName: \"kubernetes.io/projected/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-kube-api-access-vctzg\") pod \"payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg\" (UID: \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:44.747567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.747490 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25kch\" (UniqueName: \"kubernetes.io/projected/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-kube-api-access-25kch\") pod \"payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps\" (UID: \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:44.948963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:44.948879 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:45.008234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.008092 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:45.191822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.191734 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:45.232607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.232386 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps"] Nov 01 16:25:45.235405 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:25:45.235375 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccd6c08_2c16_4a6d_a2fa_18c92ee19789.slice/crio-a64ace51adabe819ef5c6ca1d5f2a196c0a73efccd4e572cf83bf80d903755d0 WatchSource:0}: Error finding container a64ace51adabe819ef5c6ca1d5f2a196c0a73efccd4e572cf83bf80d903755d0: Status 404 returned error can't find the container with id a64ace51adabe819ef5c6ca1d5f2a196c0a73efccd4e572cf83bf80d903755d0 Nov 01 16:25:45.325833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.325803 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg"] Nov 01 16:25:45.410819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.410785 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" event={"ID":"eccd6c08-2c16-4a6d-a2fa-18c92ee19789","Type":"ContainerStarted","Data":"8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371"} Nov 01 16:25:45.410984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.410847 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" event={"ID":"eccd6c08-2c16-4a6d-a2fa-18c92ee19789","Type":"ContainerStarted","Data":"a64ace51adabe819ef5c6ca1d5f2a196c0a73efccd4e572cf83bf80d903755d0"} Nov 01 16:25:45.410984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.410898 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:45.412447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.412423 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" event={"ID":"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00","Type":"ContainerStarted","Data":"6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23"} Nov 01 16:25:45.412560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.412466 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" event={"ID":"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00","Type":"ContainerStarted","Data":"e293558302cd4db40eb2502889c89dbef9ebb6ad9bbcb8fa3cc0c6f477cb5be7"} Nov 01 16:25:45.412560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.412497 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:45.433072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.433029 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" podStartSLOduration=1.4330177339999999 podStartE2EDuration="1.433017734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:45.427701452 +0000 UTC m=+2283.130867388" watchObservedRunningTime="2026-11-01 16:25:45.433017734 +0000 UTC m=+2283.136183669" Nov 01 16:25:45.443048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.442995 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" podStartSLOduration=1.44298045 podStartE2EDuration="1.44298045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:45.442837909 +0000 UTC m=+2283.146003846" watchObservedRunningTime="2026-11-01 16:25:45.44298045 +0000 UTC m=+2283.146146388" Nov 01 16:25:45.953882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:45.953844 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:46.083209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:46.083157 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:46.768017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:46.767984 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:47.660056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:47.660016 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:48.404558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.404532 4573 scope.go:122] "RemoveContainer" containerID="13c50a22ac4dcf6d5bc20f99e5bba79601484db69eb0d3593935d0db89a17279" Nov 01 16:25:48.416412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.416377 4573 scope.go:122] "RemoveContainer" containerID="92e58b37d7243fede254ca6162349a182d0d5eefb048681b3aac14060ccd5c45" Nov 01 16:25:48.428465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.428444 4573 scope.go:122] "RemoveContainer" containerID="c24bbbf925a79477aef1213d7a4c7c70ac464fcdedaf93b53dedd7252cc33e66" Nov 01 16:25:48.438868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.438851 4573 scope.go:122] "RemoveContainer" containerID="4e7b2f0c9f98b2a9898cec9e6d6172a9eb8f7c3daef2c67c559f0e08bc011215" Nov 01 16:25:48.448809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.448792 4573 scope.go:122] "RemoveContainer" containerID="b2c8fbc46dc0f1d3ecb9ba9ec7e292473d6c90b6fd201bea25181c7e66f62edf" Nov 01 16:25:48.457784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.457732 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:48.462096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.462082 4573 scope.go:122] "RemoveContainer" containerID="bf76f3636a5a866e7a7dee22343b9e5135d99614e3390ffb5b4225243b68518a" Nov 01 16:25:48.471451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.471430 4573 scope.go:122] "RemoveContainer" containerID="659ebf5de29d44d7736d4d57be52f8c38a2f52e4d4f32e00fd82610faf0767e7" Nov 01 16:25:48.481000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.480984 4573 scope.go:122] "RemoveContainer" containerID="a4067573b31634aa0164e760935a24ff0bf5ae2cbfb519062f0baaf2d585b339" Nov 01 16:25:48.490208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:48.490191 4573 scope.go:122] "RemoveContainer" containerID="5251a8619777b75f54bce17cb223a66e6cd99dc513905e9410efb61af68adf21" Nov 01 16:25:49.266067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:49.266017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:49.947436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:49.947384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:50.666906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:50.666870 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:51.366820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:51.366785 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:52.054827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:52.054791 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:52.340353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:52.340275 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:52.877141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:52.877104 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:53.212634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:53.212544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:53.645119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:53.645084 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:53.749113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:53.746137 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:53.860789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:53.859156 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:54.364802 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:25:54.364760 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:25:54.576845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:54.576807 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:55.413970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:55.413930 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:56.192630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:56.192601 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:56.426251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:56.426221 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:25:56.426620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:56.426291 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:25:56.964055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:56.964022 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:57.337541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:57.337460 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:57.448396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:57.448364 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:57.863013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:57.862977 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:57.965261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:57.965225 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:58.522945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.522903 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:58.562852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.562816 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c"] Nov 01 16:25:58.571260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.571237 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:58.575308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.575284 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c"] Nov 01 16:25:58.577244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.577224 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-104b6e\"" Nov 01 16:25:58.604077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.604047 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg"] Nov 01 16:25:58.610030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.610005 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:58.632779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.621517 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-104b6e-dockercfg-mmnmh\"" Nov 01 16:25:58.632779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.622035 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkzfc\" (UniqueName: \"kubernetes.io/projected/b33c412d-df7e-4941-acdc-47ee34d5cf0e-kube-api-access-mkzfc\") pod \"payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c\" (UID: \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:58.632779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.622083 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33c412d-df7e-4941-acdc-47ee34d5cf0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c\" (UID: \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:58.632779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.631360 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg"] Nov 01 16:25:58.632779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.632241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:58.649938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.645905 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt"] Nov 01 16:25:58.652835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.652792 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.668952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.668719 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt"] Nov 01 16:25:58.720108 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.720080 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2b94b5-serving-cert\"" Nov 01 16:25:58.723341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.723314 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52pj6\" (UniqueName: \"kubernetes.io/projected/e276535e-6c34-431a-aff5-9c5e40d32b9e-kube-api-access-52pj6\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg\" (UID: \"e276535e-6c34-431a-aff5-9c5e40d32b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:58.723512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.723363 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc6g2\" (UniqueName: \"kubernetes.io/projected/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-kube-api-access-wc6g2\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.723586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.723501 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.723666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.723655 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.723734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.723702 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkzfc\" (UniqueName: \"kubernetes.io/projected/b33c412d-df7e-4941-acdc-47ee34d5cf0e-kube-api-access-mkzfc\") pod \"payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c\" (UID: \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:58.723734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.723727 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33c412d-df7e-4941-acdc-47ee34d5cf0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c\" (UID: \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:58.723853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.723780 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.723853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.723813 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e276535e-6c34-431a-aff5-9c5e40d32b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg\" (UID: \"e276535e-6c34-431a-aff5-9c5e40d32b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:58.724537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.724516 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33c412d-df7e-4941-acdc-47ee34d5cf0e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c\" (UID: \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:58.739739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.739717 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkzfc\" (UniqueName: \"kubernetes.io/projected/b33c412d-df7e-4941-acdc-47ee34d5cf0e-kube-api-access-mkzfc\") pod \"payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c\" (UID: \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:58.800825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.800504 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:58.844845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.844804 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.845319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.845286 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.845537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.845523 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.845661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.845646 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e276535e-6c34-431a-aff5-9c5e40d32b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg\" (UID: \"e276535e-6c34-431a-aff5-9c5e40d32b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:58.845897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.845879 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pj6\" (UniqueName: \"kubernetes.io/projected/e276535e-6c34-431a-aff5-9c5e40d32b9e-kube-api-access-52pj6\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg\" (UID: \"e276535e-6c34-431a-aff5-9c5e40d32b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:58.846072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.846058 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc6g2\" (UniqueName: \"kubernetes.io/projected/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-kube-api-access-wc6g2\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.847223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.847199 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.852866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.852836 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e276535e-6c34-431a-aff5-9c5e40d32b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg\" (UID: \"e276535e-6c34-431a-aff5-9c5e40d32b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:58.890773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.890642 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc6g2\" (UniqueName: \"kubernetes.io/projected/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-kube-api-access-wc6g2\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.910848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.892070 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.910848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.892337 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:58.910848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.892707 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pj6\" (UniqueName: \"kubernetes.io/projected/e276535e-6c34-431a-aff5-9c5e40d32b9e-kube-api-access-52pj6\") pod \"payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg\" (UID: \"e276535e-6c34-431a-aff5-9c5e40d32b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:58.910848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.902626 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-metrics-certs\") pod \"maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:58.958400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.955046 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:58.958400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.958181 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:25:59.003809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:58.991063 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:25:59.319242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.319204 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c"] Nov 01 16:25:59.329816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.326303 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg"] Nov 01 16:25:59.427857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.393391 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:25:59.483939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.483882 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt"] Nov 01 16:25:59.607517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.607482 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" event={"ID":"b33c412d-df7e-4941-acdc-47ee34d5cf0e","Type":"ContainerStarted","Data":"1b6bcf96e7763cc058b705af4a42869f3e094e0ec221aa7d9b350357a553c511"} Nov 01 16:25:59.608190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.607583 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" event={"ID":"b33c412d-df7e-4941-acdc-47ee34d5cf0e","Type":"ContainerStarted","Data":"df319fb101c99a716719ec5e145263214abcbe8d07c24bb414ef3099a55d2748"} Nov 01 16:25:59.608190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.607710 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:25:59.609832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.609809 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" event={"ID":"e276535e-6c34-431a-aff5-9c5e40d32b9e","Type":"ContainerStarted","Data":"7b22368d42a4acbd0f6b6b2eec33b9e04c08880db0f10da964e95f6e50603664"} Nov 01 16:25:59.609942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.609856 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" event={"ID":"e276535e-6c34-431a-aff5-9c5e40d32b9e","Type":"ContainerStarted","Data":"6e87b0d3cf4b0fa3635ddf63ded8f999ee86beab1a1cdd2c8cec150dad61bf92"} Nov 01 16:25:59.609942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.609909 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:25:59.611469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.611445 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" event={"ID":"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e","Type":"ContainerStarted","Data":"8736c2f8943107bd3beb59008bb87e02555a52253ef3605b1936077e2572ed7e"} Nov 01 16:25:59.630283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.630223 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" podStartSLOduration=1.6302057620000001 podStartE2EDuration="1.630205762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:59.627854059 +0000 UTC m=+2297.331019996" watchObservedRunningTime="2026-11-01 16:25:59.630205762 +0000 UTC m=+2297.333371698" Nov 01 16:25:59.643379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.643325 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" podStartSLOduration=1.64330963 podStartE2EDuration="1.64330963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:25:59.642318086 +0000 UTC m=+2297.345484033" watchObservedRunningTime="2026-11-01 16:25:59.64330963 +0000 UTC m=+2297.346475568" Nov 01 16:25:59.671720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:25:59.671692 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:00.540906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:00.540869 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:00.673649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:00.673610 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" event={"ID":"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e","Type":"ContainerStarted","Data":"79f85191a88e05d9fe74c332333453e1415188a41ef14c559c3d46a632968c97"} Nov 01 16:26:00.674112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:00.673924 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:26:00.690452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:00.690401 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" podStartSLOduration=2.210687964 podStartE2EDuration="2.690389172s" totalImagesPullingTime="479.701208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:26:00.68946275 +0000 UTC m=+2298.392628697" watchObservedRunningTime="2026-11-01 16:26:00.690389172 +0000 UTC m=+2298.393555108" Nov 01 16:26:01.308940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:01.308885 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:02.055199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:02.055159 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:02.852262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:02.852204 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:03.655588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:03.655532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:04.431353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:04.431300 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:05.111938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:05.111894 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:05.762664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:05.762631 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:06.421235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:06.421198 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:07.146075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:07.146036 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:07.692993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:07.692963 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:26:07.957430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:07.957349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:08.621566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:08.621531 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:09.379791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:09.379739 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:10.079698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:10.079668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:10.675595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:10.675567 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:26:10.677002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:10.676975 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:26:10.771433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:10.771376 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:11.023662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:11.023563 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn"] Nov 01 16:26:11.028341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:11.028309 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e045478a-8r8kn"] Nov 01 16:26:11.538168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:11.538133 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:12.343355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:12.343320 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:12.463131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:12.463100 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a0a294-995d-462c-aa08-0c51bb869ae1" path="/var/lib/kubelet/pods/19a0a294-995d-462c-aa08-0c51bb869ae1/volumes" Nov 01 16:26:12.980838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:12.980804 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:13.630916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.630884 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9"] Nov 01 16:26:13.636280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.636251 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.648225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.644502 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-104b6e-serving-cert\"" Nov 01 16:26:13.651573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.651075 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9"] Nov 01 16:26:13.670555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.670518 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwps6\" (UniqueName: \"kubernetes.io/projected/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-kube-api-access-pwps6\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.670737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.670601 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-metrics-certs\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.670737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.670645 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.670737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.670728 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.680018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.679990 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:13.771902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.771867 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwps6\" (UniqueName: \"kubernetes.io/projected/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-kube-api-access-pwps6\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.772116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.771971 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-metrics-certs\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.772116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.772016 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.772243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.772139 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.772801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.772773 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.774524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.774500 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.774646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.774583 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-metrics-certs\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.779732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.779710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:13.780690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.780669 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwps6\" (UniqueName: \"kubernetes.io/projected/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-kube-api-access-pwps6\") pod \"maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:13.882858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.882724 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:13.961659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:13.961629 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:14.276211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:14.266185 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9"] Nov 01 16:26:14.541078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:14.541000 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:14.808258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:14.808212 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" event={"ID":"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe","Type":"ContainerStarted","Data":"4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b"} Nov 01 16:26:14.808731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:14.808296 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:14.808731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:14.808325 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" event={"ID":"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe","Type":"ContainerStarted","Data":"c308028acb101f8722cb82504cdb5d5a4bfaafea450d65d0d88127c941bcb95b"} Nov 01 16:26:14.828068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:14.828011 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" podStartSLOduration=1.391570537 podStartE2EDuration="1.827995694s" totalImagesPullingTime="436.425157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:26:14.823968518 +0000 UTC m=+2312.527134454" watchObservedRunningTime="2026-11-01 16:26:14.827995694 +0000 UTC m=+2312.531161631" Nov 01 16:26:15.348651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:15.348618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:15.981383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:15.981348 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:16.756773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:16.751212 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:26:16.947506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:16.947471 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:17.847258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:17.847220 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:18.499460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:18.499424 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps"] Nov 01 16:26:18.519483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:18.519454 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg"] Nov 01 16:26:18.577100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:18.577067 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:18.677578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:18.677541 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:18.822954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:18.822863 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:18.928046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:18.928013 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:19.654054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:19.654017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:20.347796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:20.347736 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:20.822268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:20.822242 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:26:21.079197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.079120 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:21.197326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.197293 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2"] Nov 01 16:26:21.204720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.204683 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.208541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.208514 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:26:21.208801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.208778 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:26:21.209652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.209626 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2"] Nov 01 16:26:21.265166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.265131 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.265603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.265577 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqsd5\" (UniqueName: \"kubernetes.io/projected/b986943c-9407-4ddd-b125-1f99a1ef2eb9-kube-api-access-bqsd5\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.265723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.265698 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.367110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.367084 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqsd5\" (UniqueName: \"kubernetes.io/projected/b986943c-9407-4ddd-b125-1f99a1ef2eb9-kube-api-access-bqsd5\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.367572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.367137 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.367572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.367164 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.367904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.367885 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.369467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.369447 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.374954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.374931 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqsd5\" (UniqueName: \"kubernetes.io/projected/b986943c-9407-4ddd-b125-1f99a1ef2eb9-kube-api-access-bqsd5\") pod \"maas-api-revoke-sub-7e96ce66-2gxk2\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.534522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.534485 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:21.764540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.764456 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:21.857274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.854941 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2"] Nov 01 16:26:21.866473 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:26:21.866435 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb986943c_9407_4ddd_b125_1f99a1ef2eb9.slice/crio-3b9ece688499a6f76294b99c9f17a7fa67cc6675812912de3fbb77adf8fd1047 WatchSource:0}: Error finding container 3b9ece688499a6f76294b99c9f17a7fa67cc6675812912de3fbb77adf8fd1047: Status 404 returned error can't find the container with id 3b9ece688499a6f76294b99c9f17a7fa67cc6675812912de3fbb77adf8fd1047 Nov 01 16:26:21.912601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:21.912560 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" event={"ID":"b986943c-9407-4ddd-b125-1f99a1ef2eb9","Type":"ContainerStarted","Data":"3b9ece688499a6f76294b99c9f17a7fa67cc6675812912de3fbb77adf8fd1047"} Nov 01 16:26:22.452073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:22.452037 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:22.925210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:22.925176 4573 generic.go:356] "Generic (PLEG): container finished" podID="b986943c-9407-4ddd-b125-1f99a1ef2eb9" containerID="15c1ac11d0da1767b5775b31341cedd985c2d1505f4ff570131ee74cf0ae6d2b" exitCode=0 Nov 01 16:26:22.925414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:22.925369 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" event={"ID":"b986943c-9407-4ddd-b125-1f99a1ef2eb9","Type":"ContainerDied","Data":"15c1ac11d0da1767b5775b31341cedd985c2d1505f4ff570131ee74cf0ae6d2b"} Nov 01 16:26:23.184938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.184861 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:23.500814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.500690 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" podUID="eccd6c08-2c16-4a6d-a2fa-18c92ee19789" containerName="payload-processing" containerID="cri-o://8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371" gracePeriod=25 Nov 01 16:26:23.520246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.520182 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" podUID="c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00" containerName="payload-pre-processing" containerID="cri-o://6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23" gracePeriod=25 Nov 01 16:26:23.831395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.831364 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:26:23.870522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.870497 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:26:23.891277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.891109 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:23.917083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.917052 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-kube-api-access-25kch\" (UniqueName: \"kubernetes.io/projected/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-kube-api-access-25kch\") pod \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\" (UID: \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\") " Nov 01 16:26:23.917270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.917167 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-plugins-config-volume\") pod \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\" (UID: \"eccd6c08-2c16-4a6d-a2fa-18c92ee19789\") " Nov 01 16:26:23.917399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.917379 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-kube-api-access-vctzg\" (UniqueName: \"kubernetes.io/projected/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-kube-api-access-vctzg\") pod \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\" (UID: \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\") " Nov 01 16:26:23.917512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.917492 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-plugins-config-volume\") pod \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\" (UID: \"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00\") " Nov 01 16:26:23.917593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.917567 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-plugins-config-volume" pod "eccd6c08-2c16-4a6d-a2fa-18c92ee19789" (UID: "eccd6c08-2c16-4a6d-a2fa-18c92ee19789"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:26:23.918116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.918094 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-plugins-config-volume" pod "c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00" (UID: "c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:26:23.918316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.918300 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:23.918387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.918319 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:23.919917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.919884 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-kube-api-access-25kch" pod "eccd6c08-2c16-4a6d-a2fa-18c92ee19789" (UID: "eccd6c08-2c16-4a6d-a2fa-18c92ee19789"). InnerVolumeSpecName "kube-api-access-25kch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:23.920020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.919966 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-kube-api-access-vctzg" pod "c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00" (UID: "c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00"). InnerVolumeSpecName "kube-api-access-vctzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:23.945943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.940951 4573 generic.go:356] "Generic (PLEG): container finished" podID="eccd6c08-2c16-4a6d-a2fa-18c92ee19789" containerID="8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371" exitCode=0 Nov 01 16:26:23.945943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.941076 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" event={"ID":"eccd6c08-2c16-4a6d-a2fa-18c92ee19789","Type":"ContainerDied","Data":"8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371"} Nov 01 16:26:23.945943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.941132 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" event={"ID":"eccd6c08-2c16-4a6d-a2fa-18c92ee19789","Type":"ContainerDied","Data":"a64ace51adabe819ef5c6ca1d5f2a196c0a73efccd4e572cf83bf80d903755d0"} Nov 01 16:26:23.945943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.941169 4573 scope.go:122] "RemoveContainer" containerID="8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371" Nov 01 16:26:23.945943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.941343 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps" Nov 01 16:26:23.981741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.978280 4573 generic.go:356] "Generic (PLEG): container finished" podID="c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00" containerID="6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23" exitCode=0 Nov 01 16:26:23.982392 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.981121 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" Nov 01 16:26:23.993637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.981689 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" event={"ID":"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00","Type":"ContainerDied","Data":"6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23"} Nov 01 16:26:23.993637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.993494 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg" event={"ID":"c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00","Type":"ContainerDied","Data":"e293558302cd4db40eb2502889c89dbef9ebb6ad9bbcb8fa3cc0c6f477cb5be7"} Nov 01 16:26:23.997678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:23.997626 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps"] Nov 01 16:26:24.004974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.004949 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-758df97487-4d6ps"] Nov 01 16:26:24.018700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.018683 4573 scope.go:122] "RemoveContainer" containerID="8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371" Nov 01 16:26:24.019986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.019943 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25kch\" (UniqueName: \"kubernetes.io/projected/eccd6c08-2c16-4a6d-a2fa-18c92ee19789-kube-api-access-25kch\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:24.019986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.019963 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vctzg\" (UniqueName: \"kubernetes.io/projected/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00-kube-api-access-vctzg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:24.020151 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:26:24.020035 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371\": container with ID starting with 8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371 not found: ID does not exist" containerID="8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371" Nov 01 16:26:24.020151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.020058 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371"} err="failed to get container status \"8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371\": rpc error: code = NotFound desc = could not find container \"8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371\": container with ID starting with 8ab73aa4a957308334be9709a4eedc8db406f368526dbfd29eceeb771a7d2371 not found: ID does not exist" Nov 01 16:26:24.020151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.020073 4573 scope.go:122] "RemoveContainer" containerID="6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23" Nov 01 16:26:24.033014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.031883 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg"] Nov 01 16:26:24.036174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.036145 4573 scope.go:122] "RemoveContainer" containerID="6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23" Nov 01 16:26:24.036643 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:26:24.036611 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23\": container with ID starting with 6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23 not found: ID does not exist" containerID="6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23" Nov 01 16:26:24.036721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.036656 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23"} err="failed to get container status \"6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23\": rpc error: code = NotFound desc = could not find container \"6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23\": container with ID starting with 6adeaf7fb211d0deba6cda365061a56b3d89130cb1d5b189282a83c41919fa23 not found: ID does not exist" Nov 01 16:26:24.037063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.037043 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2b94b5-75f6495c99-c5ljg"] Nov 01 16:26:24.153657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.153631 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:24.223957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.223915 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-openshift-ca\") pod \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " Nov 01 16:26:24.224482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.224451 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-openshift-ca" pod "b986943c-9407-4ddd-b125-1f99a1ef2eb9" (UID: "b986943c-9407-4ddd-b125-1f99a1ef2eb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:26:24.224681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.224470 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b986943c-9407-4ddd-b125-1f99a1ef2eb9-kube-api-access-bqsd5\" (UniqueName: \"kubernetes.io/projected/b986943c-9407-4ddd-b125-1f99a1ef2eb9-kube-api-access-bqsd5\") pod \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " Nov 01 16:26:24.224853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.224831 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-xks-ca\") pod \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\" (UID: \"b986943c-9407-4ddd-b125-1f99a1ef2eb9\") " Nov 01 16:26:24.225934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.225913 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:24.228221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.228186 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b986943c-9407-4ddd-b125-1f99a1ef2eb9-kube-api-access-bqsd5" pod "b986943c-9407-4ddd-b125-1f99a1ef2eb9" (UID: "b986943c-9407-4ddd-b125-1f99a1ef2eb9"). InnerVolumeSpecName "kube-api-access-bqsd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:24.228888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.228866 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-xks-ca" pod "b986943c-9407-4ddd-b125-1f99a1ef2eb9" (UID: "b986943c-9407-4ddd-b125-1f99a1ef2eb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:26:24.328088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.328014 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqsd5\" (UniqueName: \"kubernetes.io/projected/b986943c-9407-4ddd-b125-1f99a1ef2eb9-kube-api-access-bqsd5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:24.328088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.328038 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b986943c-9407-4ddd-b125-1f99a1ef2eb9-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:24.365198 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:26:24.365168 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:26:24.460799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.460768 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00" path="/var/lib/kubelet/pods/c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00/volumes" Nov 01 16:26:24.461275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.461259 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eccd6c08-2c16-4a6d-a2fa-18c92ee19789" path="/var/lib/kubelet/pods/eccd6c08-2c16-4a6d-a2fa-18c92ee19789/volumes" Nov 01 16:26:24.569695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.569661 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:24.992422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.992393 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" Nov 01 16:26:24.992422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.992413 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2" event={"ID":"b986943c-9407-4ddd-b125-1f99a1ef2eb9","Type":"ContainerDied","Data":"3b9ece688499a6f76294b99c9f17a7fa67cc6675812912de3fbb77adf8fd1047"} Nov 01 16:26:24.992717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:24.992457 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b9ece688499a6f76294b99c9f17a7fa67cc6675812912de3fbb77adf8fd1047" Nov 01 16:26:25.193970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:25.193916 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:25.817216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:25.817183 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:26.480954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:26.480915 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:27.155680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:27.155642 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:27.768482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:27.768445 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:28.540792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:28.540737 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:29.551209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:29.551171 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:30.023498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:30.023463 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r"] Nov 01 16:26:30.027649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:30.027624 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c25547-np89r"] Nov 01 16:26:30.258881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:30.258842 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:30.460836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:30.460802 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62e9e631-16f2-49f0-92ac-4596545b643c" path="/var/lib/kubelet/pods/62e9e631-16f2-49f0-92ac-4596545b643c/volumes" Nov 01 16:26:30.878615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:30.878581 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:31.502743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.502710 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2"] Nov 01 16:26:31.503637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503613 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b986943c-9407-4ddd-b125-1f99a1ef2eb9" containerName="revoke-keys" Nov 01 16:26:31.503637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503636 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b986943c-9407-4ddd-b125-1f99a1ef2eb9" containerName="revoke-keys" Nov 01 16:26:31.503897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503661 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00" containerName="payload-pre-processing" Nov 01 16:26:31.503897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503667 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00" containerName="payload-pre-processing" Nov 01 16:26:31.503897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503707 4573 cpu_manager.go:414] "containerMap: removing container" podUID="eccd6c08-2c16-4a6d-a2fa-18c92ee19789" containerName="payload-processing" Nov 01 16:26:31.503897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503712 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eccd6c08-2c16-4a6d-a2fa-18c92ee19789" containerName="payload-processing" Nov 01 16:26:31.504045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503920 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b986943c-9407-4ddd-b125-1f99a1ef2eb9" containerName="revoke-keys" Nov 01 16:26:31.504045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503939 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5a133d0-ffc6-4e4b-ba1c-a36d09b49a00" containerName="payload-pre-processing" Nov 01 16:26:31.504045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.503953 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="eccd6c08-2c16-4a6d-a2fa-18c92ee19789" containerName="payload-processing" Nov 01 16:26:31.509266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.509224 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.515394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.515369 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:26:31.515623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.515606 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:26:31.516285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.516260 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2"] Nov 01 16:26:31.669139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.669098 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.669734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.669685 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtjt\" (UniqueName: \"kubernetes.io/projected/ce15495c-2781-485a-b362-7c71f5540a67-kube-api-access-trtjt\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.670154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.670135 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.771581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.771495 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.771776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.771660 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.771878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.771849 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtjt\" (UniqueName: \"kubernetes.io/projected/ce15495c-2781-485a-b362-7c71f5540a67-kube-api-access-trtjt\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.772215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.772190 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.774199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.774178 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.780345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.780319 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtjt\" (UniqueName: \"kubernetes.io/projected/ce15495c-2781-485a-b362-7c71f5540a67-kube-api-access-trtjt\") pod \"maas-api-revoke-sub-0635104d-4f9z2\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.841197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.841160 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:31.926385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:31.926355 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:32.128973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:32.128937 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:32.144004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:32.139186 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2"] Nov 01 16:26:32.670165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:32.670129 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:32.847025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:32.846941 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c"] Nov 01 16:26:32.866883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:32.865064 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg"] Nov 01 16:26:32.912808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:32.912214 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:33.006836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:33.006801 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:33.066316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:33.066272 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" event={"ID":"ce15495c-2781-485a-b362-7c71f5540a67","Type":"ContainerStarted","Data":"1399703c91306f69adf7eef740b496adcece8e4b6a26d1586944e83a431f9d7d"} Nov 01 16:26:33.066620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:33.066335 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" event={"ID":"ce15495c-2781-485a-b362-7c71f5540a67","Type":"ContainerStarted","Data":"34d7e4b9b392d5c739294d5e68fe234099129e03aa7eabd316e162802fb1c51b"} Nov 01 16:26:33.087129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:33.087085 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" podStartSLOduration=2.087070275 podStartE2EDuration="2.087070275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:26:33.086131116 +0000 UTC m=+2330.789297055" watchObservedRunningTime="2026-11-01 16:26:33.087070275 +0000 UTC m=+2330.790236210" Nov 01 16:26:33.090451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:33.090424 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:33.614174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:33.614138 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:34.079885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.079693 4573 generic.go:356] "Generic (PLEG): container finished" podID="ce15495c-2781-485a-b362-7c71f5540a67" containerID="1399703c91306f69adf7eef740b496adcece8e4b6a26d1586944e83a431f9d7d" exitCode=0 Nov 01 16:26:34.080362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.080054 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" event={"ID":"ce15495c-2781-485a-b362-7c71f5540a67","Type":"ContainerDied","Data":"1399703c91306f69adf7eef740b496adcece8e4b6a26d1586944e83a431f9d7d"} Nov 01 16:26:34.104560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.104534 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:34.202125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.202094 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:34.236786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.236722 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l"] Nov 01 16:26:34.241677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.241655 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:34.245463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.245423 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2b94b5-dockercfg-2fq72\"" Nov 01 16:26:34.245610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.245490 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2b94b5\"" Nov 01 16:26:34.249998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.249976 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l"] Nov 01 16:26:34.341932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.341830 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d866f151-f5b8-4f65-b376-24e6c0da561e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l\" (UID: \"d866f151-f5b8-4f65-b376-24e6c0da561e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:34.342129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.342051 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx65m\" (UniqueName: \"kubernetes.io/projected/d866f151-f5b8-4f65-b376-24e6c0da561e-kube-api-access-tx65m\") pod \"payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l\" (UID: \"d866f151-f5b8-4f65-b376-24e6c0da561e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:34.460776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.454048 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d866f151-f5b8-4f65-b376-24e6c0da561e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l\" (UID: \"d866f151-f5b8-4f65-b376-24e6c0da561e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:34.460776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.454884 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d866f151-f5b8-4f65-b376-24e6c0da561e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l\" (UID: \"d866f151-f5b8-4f65-b376-24e6c0da561e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:34.460776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.455142 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx65m\" (UniqueName: \"kubernetes.io/projected/d866f151-f5b8-4f65-b376-24e6c0da561e-kube-api-access-tx65m\") pod \"payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l\" (UID: \"d866f151-f5b8-4f65-b376-24e6c0da561e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:34.515772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.515028 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx65m\" (UniqueName: \"kubernetes.io/projected/d866f151-f5b8-4f65-b376-24e6c0da561e-kube-api-access-tx65m\") pod \"payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l\" (UID: \"d866f151-f5b8-4f65-b376-24e6c0da561e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:34.560299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.560267 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:34.776482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.776424 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:34.887739 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:26:34.887703 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd866f151_f5b8_4f65_b376_24e6c0da561e.slice/crio-de94dd2196e0ed6fd390bbdb1ecd42556a130cab680435510358c5c44187f7c3 WatchSource:0}: Error finding container de94dd2196e0ed6fd390bbdb1ecd42556a130cab680435510358c5c44187f7c3: Status 404 returned error can't find the container with id de94dd2196e0ed6fd390bbdb1ecd42556a130cab680435510358c5c44187f7c3 Nov 01 16:26:34.888462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:34.888420 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l"] Nov 01 16:26:35.090923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.090884 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" event={"ID":"d866f151-f5b8-4f65-b376-24e6c0da561e","Type":"ContainerStarted","Data":"63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea"} Nov 01 16:26:35.091368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.090976 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" event={"ID":"d866f151-f5b8-4f65-b376-24e6c0da561e","Type":"ContainerStarted","Data":"de94dd2196e0ed6fd390bbdb1ecd42556a130cab680435510358c5c44187f7c3"} Nov 01 16:26:35.091368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.091008 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:35.114854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.114791 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" podStartSLOduration=1.114767374 podStartE2EDuration="1.114767374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:26:35.107712495 +0000 UTC m=+2332.810878432" watchObservedRunningTime="2026-11-01 16:26:35.114767374 +0000 UTC m=+2332.817933307" Nov 01 16:26:35.300994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.300963 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:35.474803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.474705 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce15495c-2781-485a-b362-7c71f5540a67-kube-api-access-trtjt\" (UniqueName: \"kubernetes.io/projected/ce15495c-2781-485a-b362-7c71f5540a67-kube-api-access-trtjt\") pod \"ce15495c-2781-485a-b362-7c71f5540a67\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " Nov 01 16:26:35.474967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.474839 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-openshift-ca\") pod \"ce15495c-2781-485a-b362-7c71f5540a67\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " Nov 01 16:26:35.474967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.474880 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-xks-ca\") pod \"ce15495c-2781-485a-b362-7c71f5540a67\" (UID: \"ce15495c-2781-485a-b362-7c71f5540a67\") " Nov 01 16:26:35.475206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.475175 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-openshift-ca" pod "ce15495c-2781-485a-b362-7c71f5540a67" (UID: "ce15495c-2781-485a-b362-7c71f5540a67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:26:35.475415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.475396 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:35.477067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.477040 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce15495c-2781-485a-b362-7c71f5540a67-kube-api-access-trtjt" pod "ce15495c-2781-485a-b362-7c71f5540a67" (UID: "ce15495c-2781-485a-b362-7c71f5540a67"). InnerVolumeSpecName "kube-api-access-trtjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:35.477067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.477056 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-xks-ca" pod "ce15495c-2781-485a-b362-7c71f5540a67" (UID: "ce15495c-2781-485a-b362-7c71f5540a67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:26:35.532873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.532839 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:35.576694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.576665 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trtjt\" (UniqueName: \"kubernetes.io/projected/ce15495c-2781-485a-b362-7c71f5540a67-kube-api-access-trtjt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:35.576694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:35.576692 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce15495c-2781-485a-b362-7c71f5540a67-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:36.100921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:36.100888 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" Nov 01 16:26:36.101308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:36.100919 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2" event={"ID":"ce15495c-2781-485a-b362-7c71f5540a67","Type":"ContainerDied","Data":"34d7e4b9b392d5c739294d5e68fe234099129e03aa7eabd316e162802fb1c51b"} Nov 01 16:26:36.101308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:36.100974 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34d7e4b9b392d5c739294d5e68fe234099129e03aa7eabd316e162802fb1c51b" Nov 01 16:26:36.183223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:36.183180 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:36.888436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:36.888395 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:37.575541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:37.575474 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:37.847803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:37.847683 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" podUID="b33c412d-df7e-4941-acdc-47ee34d5cf0e" containerName="payload-processing" containerID="cri-o://1b6bcf96e7763cc058b705af4a42869f3e094e0ec221aa7d9b350357a553c511" gracePeriod=25 Nov 01 16:26:37.865829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:37.865796 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" podUID="e276535e-6c34-431a-aff5-9c5e40d32b9e" containerName="payload-pre-processing" containerID="cri-o://7b22368d42a4acbd0f6b6b2eec33b9e04c08880db0f10da964e95f6e50603664" gracePeriod=25 Nov 01 16:26:38.130943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.130908 4573 generic.go:356] "Generic (PLEG): container finished" podID="b33c412d-df7e-4941-acdc-47ee34d5cf0e" containerID="1b6bcf96e7763cc058b705af4a42869f3e094e0ec221aa7d9b350357a553c511" exitCode=0 Nov 01 16:26:38.131137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.131038 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" event={"ID":"b33c412d-df7e-4941-acdc-47ee34d5cf0e","Type":"ContainerDied","Data":"1b6bcf96e7763cc058b705af4a42869f3e094e0ec221aa7d9b350357a553c511"} Nov 01 16:26:38.133309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.133284 4573 generic.go:356] "Generic (PLEG): container finished" podID="e276535e-6c34-431a-aff5-9c5e40d32b9e" containerID="7b22368d42a4acbd0f6b6b2eec33b9e04c08880db0f10da964e95f6e50603664" exitCode=0 Nov 01 16:26:38.133616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.133595 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" event={"ID":"e276535e-6c34-431a-aff5-9c5e40d32b9e","Type":"ContainerDied","Data":"7b22368d42a4acbd0f6b6b2eec33b9e04c08880db0f10da964e95f6e50603664"} Nov 01 16:26:38.183786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.183722 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:26:38.217265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.217233 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b33c412d-df7e-4941-acdc-47ee34d5cf0e-kube-api-access-mkzfc\" (UniqueName: \"kubernetes.io/projected/b33c412d-df7e-4941-acdc-47ee34d5cf0e-kube-api-access-mkzfc\") pod \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\" (UID: \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\") " Nov 01 16:26:38.217472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.217454 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b33c412d-df7e-4941-acdc-47ee34d5cf0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33c412d-df7e-4941-acdc-47ee34d5cf0e-plugins-config-volume\") pod \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\" (UID: \"b33c412d-df7e-4941-acdc-47ee34d5cf0e\") " Nov 01 16:26:38.217919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.217889 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b33c412d-df7e-4941-acdc-47ee34d5cf0e-plugins-config-volume" pod "b33c412d-df7e-4941-acdc-47ee34d5cf0e" (UID: "b33c412d-df7e-4941-acdc-47ee34d5cf0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:26:38.218149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.218133 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b33c412d-df7e-4941-acdc-47ee34d5cf0e-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:38.219907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.219878 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b33c412d-df7e-4941-acdc-47ee34d5cf0e-kube-api-access-mkzfc" pod "b33c412d-df7e-4941-acdc-47ee34d5cf0e" (UID: "b33c412d-df7e-4941-acdc-47ee34d5cf0e"). InnerVolumeSpecName "kube-api-access-mkzfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:38.250758 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.250720 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:26:38.270272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.270241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:38.345428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.344208 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e276535e-6c34-431a-aff5-9c5e40d32b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e276535e-6c34-431a-aff5-9c5e40d32b9e-plugins-config-volume\") pod \"e276535e-6c34-431a-aff5-9c5e40d32b9e\" (UID: \"e276535e-6c34-431a-aff5-9c5e40d32b9e\") " Nov 01 16:26:38.345428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.344569 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e276535e-6c34-431a-aff5-9c5e40d32b9e-kube-api-access-52pj6\" (UniqueName: \"kubernetes.io/projected/e276535e-6c34-431a-aff5-9c5e40d32b9e-kube-api-access-52pj6\") pod \"e276535e-6c34-431a-aff5-9c5e40d32b9e\" (UID: \"e276535e-6c34-431a-aff5-9c5e40d32b9e\") " Nov 01 16:26:38.345822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.345513 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkzfc\" (UniqueName: \"kubernetes.io/projected/b33c412d-df7e-4941-acdc-47ee34d5cf0e-kube-api-access-mkzfc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:38.346399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.346345 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e276535e-6c34-431a-aff5-9c5e40d32b9e-plugins-config-volume" pod "e276535e-6c34-431a-aff5-9c5e40d32b9e" (UID: "e276535e-6c34-431a-aff5-9c5e40d32b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:26:38.364919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.362807 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e276535e-6c34-431a-aff5-9c5e40d32b9e-kube-api-access-52pj6" pod "e276535e-6c34-431a-aff5-9c5e40d32b9e" (UID: "e276535e-6c34-431a-aff5-9c5e40d32b9e"). InnerVolumeSpecName "kube-api-access-52pj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:26:38.447792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.447699 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52pj6\" (UniqueName: \"kubernetes.io/projected/e276535e-6c34-431a-aff5-9c5e40d32b9e-kube-api-access-52pj6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:38.447792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.447728 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e276535e-6c34-431a-aff5-9c5e40d32b9e-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:26:38.950050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:38.950018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:39.146886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.146774 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" event={"ID":"e276535e-6c34-431a-aff5-9c5e40d32b9e","Type":"ContainerDied","Data":"6e87b0d3cf4b0fa3635ddf63ded8f999ee86beab1a1cdd2c8cec150dad61bf92"} Nov 01 16:26:39.146886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.146816 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg" Nov 01 16:26:39.147133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.146902 4573 scope.go:122] "RemoveContainer" containerID="7b22368d42a4acbd0f6b6b2eec33b9e04c08880db0f10da964e95f6e50603664" Nov 01 16:26:39.155505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.155330 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" event={"ID":"b33c412d-df7e-4941-acdc-47ee34d5cf0e","Type":"ContainerDied","Data":"df319fb101c99a716719ec5e145263214abcbe8d07c24bb414ef3099a55d2748"} Nov 01 16:26:39.155505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.155350 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c" Nov 01 16:26:39.168350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.168215 4573 scope.go:122] "RemoveContainer" containerID="1b6bcf96e7763cc058b705af4a42869f3e094e0ec221aa7d9b350357a553c511" Nov 01 16:26:39.170466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.170442 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg"] Nov 01 16:26:39.174763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.174730 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-104b6e-6dfb84c587-8rgrg"] Nov 01 16:26:39.184645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.184623 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c"] Nov 01 16:26:39.186438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.186419 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-6dd5796b49-86v9c"] Nov 01 16:26:39.681772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:39.673317 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:40.350326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:40.350285 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:40.463966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:40.463928 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b33c412d-df7e-4941-acdc-47ee34d5cf0e" path="/var/lib/kubelet/pods/b33c412d-df7e-4941-acdc-47ee34d5cf0e/volumes" Nov 01 16:26:40.464644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:40.464622 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e276535e-6c34-431a-aff5-9c5e40d32b9e" path="/var/lib/kubelet/pods/e276535e-6c34-431a-aff5-9c5e40d32b9e/volumes" Nov 01 16:26:41.043941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:41.043904 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:41.662463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:41.662416 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:42.284922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:42.284877 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:42.971905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:42.971855 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:43.251679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:43.251569 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:43.862635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:43.862602 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:44.546621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:44.546586 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:45.273282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:45.273233 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:45.709446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:45.709410 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:46.103655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:46.103574 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:26:46.194303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:46.194273 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:47.073140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:47.073103 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:47.835926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:47.835890 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:48.566111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:48.566075 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:48.658538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:48.653905 4573 scope.go:122] "RemoveContainer" containerID="a31f0699772a6b8cbb6df10017c32c688954a850410b24dc52b9ca03442833d3" Nov 01 16:26:48.684698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:48.684670 4573 scope.go:122] "RemoveContainer" containerID="6eecd7372de6777e4dfaf1e342665f84a95e07160538bb88405f3c0903923299" Nov 01 16:26:48.697826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:48.697798 4573 scope.go:122] "RemoveContainer" containerID="d09b0a3cdfafa12f4216c94d9d0716b3372295157b5c7603e4106be5d7ae1dd7" Nov 01 16:26:49.272217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.272180 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:49.364035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.364007 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:49.365445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.365421 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz"] Nov 01 16:26:49.366194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366179 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ce15495c-2781-485a-b362-7c71f5540a67" containerName="revoke-keys" Nov 01 16:26:49.366245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366195 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce15495c-2781-485a-b362-7c71f5540a67" containerName="revoke-keys" Nov 01 16:26:49.366245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366206 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e276535e-6c34-431a-aff5-9c5e40d32b9e" containerName="payload-pre-processing" Nov 01 16:26:49.366245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366211 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e276535e-6c34-431a-aff5-9c5e40d32b9e" containerName="payload-pre-processing" Nov 01 16:26:49.366348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366251 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b33c412d-df7e-4941-acdc-47ee34d5cf0e" containerName="payload-processing" Nov 01 16:26:49.366348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366256 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b33c412d-df7e-4941-acdc-47ee34d5cf0e" containerName="payload-processing" Nov 01 16:26:49.366413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366377 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b33c412d-df7e-4941-acdc-47ee34d5cf0e" containerName="payload-processing" Nov 01 16:26:49.366413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366388 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce15495c-2781-485a-b362-7c71f5540a67" containerName="revoke-keys" Nov 01 16:26:49.366413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.366396 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e276535e-6c34-431a-aff5-9c5e40d32b9e" containerName="payload-pre-processing" Nov 01 16:26:49.372809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.372169 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:49.376323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.376294 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-104b6e\"" Nov 01 16:26:49.376470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.376453 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-104b6e-dockercfg-b7rf5\"" Nov 01 16:26:49.378946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.378922 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz"] Nov 01 16:26:49.456828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.456792 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc629\" (UniqueName: \"kubernetes.io/projected/ba2255ef-523d-4c04-831c-5c07035a96ee-kube-api-access-mc629\") pod \"payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz\" (UID: \"ba2255ef-523d-4c04-831c-5c07035a96ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:49.457021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.456855 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba2255ef-523d-4c04-831c-5c07035a96ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz\" (UID: \"ba2255ef-523d-4c04-831c-5c07035a96ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:49.571191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.571096 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc629\" (UniqueName: \"kubernetes.io/projected/ba2255ef-523d-4c04-831c-5c07035a96ee-kube-api-access-mc629\") pod \"payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz\" (UID: \"ba2255ef-523d-4c04-831c-5c07035a96ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:49.571191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.571167 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba2255ef-523d-4c04-831c-5c07035a96ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz\" (UID: \"ba2255ef-523d-4c04-831c-5c07035a96ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:49.572669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.572191 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba2255ef-523d-4c04-831c-5c07035a96ee-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz\" (UID: \"ba2255ef-523d-4c04-831c-5c07035a96ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:49.674405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.674367 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc629\" (UniqueName: \"kubernetes.io/projected/ba2255ef-523d-4c04-831c-5c07035a96ee-kube-api-access-mc629\") pod \"payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz\" (UID: \"ba2255ef-523d-4c04-831c-5c07035a96ee\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:49.698449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.698411 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:49.855062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.854976 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:49.994938 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:26:49.994905 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2255ef_523d_4c04_831c_5c07035a96ee.slice/crio-4f120643a8b13e2a26460f05efd330fb8e2a0486da84636887b01ac769f0d412 WatchSource:0}: Error finding container 4f120643a8b13e2a26460f05efd330fb8e2a0486da84636887b01ac769f0d412: Status 404 returned error can't find the container with id 4f120643a8b13e2a26460f05efd330fb8e2a0486da84636887b01ac769f0d412 Nov 01 16:26:49.996325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:49.996299 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz"] Nov 01 16:26:50.252238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:50.252206 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" event={"ID":"ba2255ef-523d-4c04-831c-5c07035a96ee","Type":"ContainerStarted","Data":"8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f"} Nov 01 16:26:50.252434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:50.252259 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" event={"ID":"ba2255ef-523d-4c04-831c-5c07035a96ee","Type":"ContainerStarted","Data":"4f120643a8b13e2a26460f05efd330fb8e2a0486da84636887b01ac769f0d412"} Nov 01 16:26:50.252434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:50.252297 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:26:50.271055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:50.271000 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" podStartSLOduration=1.270982995 podStartE2EDuration="1.270982995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:26:50.267216031 +0000 UTC m=+2347.970381970" watchObservedRunningTime="2026-11-01 16:26:50.270982995 +0000 UTC m=+2347.974148932" Nov 01 16:26:50.692258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:50.692227 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:51.462879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:51.462828 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:52.250326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:52.250294 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:52.971908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:52.971870 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:53.672938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:53.672902 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:54.365144 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:26:54.365110 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:26:54.470843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:54.470730 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:55.185336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:55.185307 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:56.053666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:56.053627 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:56.761898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:56.761866 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:57.087694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:57.087598 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:57.679356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:57.679321 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:58.449478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:58.449447 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:58.852666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:58.852588 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:26:59.316956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:26:59.316919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:00.156602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:00.156574 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:00.856807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:00.856766 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:01.263560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:01.263482 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:27:01.567628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:01.567533 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr"] Nov 01 16:27:01.567848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:01.567819 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" containerID="cri-o://f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7" gracePeriod=30 Nov 01 16:27:01.657514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:01.657478 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:02.474013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:02.473979 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:03.311977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:03.311943 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:03.980259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:03.980217 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:04.761040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:04.761010 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:04.862854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:04.862811 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:05.255170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:05.255133 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:05.564884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:05.563521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:05.877179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:05.877147 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:06.564365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.564329 4573 patch_prober.go:29] interesting pod/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:27:06.564785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.564392 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:27:06.653768 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.653728 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:06.902946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.902921 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 16:27:06.942303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.942271 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-data\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.942565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.942339 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-podinfo\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.942565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.942384 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-envoy\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.942565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.942490 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istiod-ca-cert\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.942565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.942523 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-token\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.943302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.943265 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-data" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.943837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.943797 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istiod-ca-cert" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:06.943989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.943963 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-socket\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.944064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.944031 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-kube-api-access-6r4jf\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-kube-api-access-6r4jf\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.944735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.944708 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-socket" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.944877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.944814 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-credential-socket\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.944877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.944845 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-certs\") pod \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\" (UID: \"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b\") " Nov 01 16:27:06.945328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.945278 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-credential-socket" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.945658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.945621 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-certs" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.945870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.945847 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.945994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.945878 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.945994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.945895 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.945994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.945911 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.945994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.945924 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:06.948798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.948726 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-podinfo" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:27:06.948907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.948796 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-envoy" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:06.948907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.948834 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-kube-api-access-6r4jf" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "kube-api-access-6r4jf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:06.949029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:06.948986 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-token" pod "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" (UID: "e9ad92a9-1bc0-4c24-a549-d76ab7e8419b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:07.047957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.047924 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:07.048119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.047977 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r4jf\" (UniqueName: \"kubernetes.io/projected/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-kube-api-access-6r4jf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:07.048119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.047988 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:07.048119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.047998 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:07.363737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.363699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:07.540514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.540478 4573 generic.go:356] "Generic (PLEG): container finished" podID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerID="f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7" exitCode=0 Nov 01 16:27:07.540697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.540525 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" Nov 01 16:27:07.540697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.540544 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" event={"ID":"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b","Type":"ContainerDied","Data":"f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7"} Nov 01 16:27:07.540697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.540596 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr" event={"ID":"e9ad92a9-1bc0-4c24-a549-d76ab7e8419b","Type":"ContainerDied","Data":"b3603f555f07fa8fa950948b34d704cacdb4533f913e8ed859c8d2d1cba439ea"} Nov 01 16:27:07.540697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.540626 4573 scope.go:122] "RemoveContainer" containerID="f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7" Nov 01 16:27:07.552640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.552618 4573 scope.go:122] "RemoveContainer" containerID="f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7" Nov 01 16:27:07.552953 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:27:07.552927 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7\": container with ID starting with f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7 not found: ID does not exist" containerID="f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7" Nov 01 16:27:07.553015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.552967 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7"} err="failed to get container status \"f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7\": rpc error: code = NotFound desc = could not find container \"f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7\": container with ID starting with f41ab472ee156b0dfd2449e9ec300b845060721c8486c8e7193fe45b939d8ad7 not found: ID does not exist" Nov 01 16:27:07.566949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.566928 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr"] Nov 01 16:27:07.571337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:07.571316 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-vr5hr"] Nov 01 16:27:08.348055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:08.348023 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:08.461347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:08.461315 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" path="/var/lib/kubelet/pods/e9ad92a9-1bc0-4c24-a549-d76ab7e8419b/volumes" Nov 01 16:27:08.607567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:08.607532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:08.706358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:08.706324 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:09.146413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:09.146381 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:09.901393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:09.901352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:09.988563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:09.988528 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:10.429993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:10.429939 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:11.811434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:11.811394 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:13.128456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:13.128423 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:13.893828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:13.893797 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:14.555014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:14.554974 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:14.811813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:14.811702 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:15.214452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.214417 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws"] Nov 01 16:27:15.215257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.215241 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" Nov 01 16:27:15.215316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.215258 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" Nov 01 16:27:15.215416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.215406 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ad92a9-1bc0-4c24-a549-d76ab7e8419b" containerName="istio-proxy" Nov 01 16:27:15.219891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.219866 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.224017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.223994 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:27:15.224241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.224219 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:27:15.226786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.226720 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws"] Nov 01 16:27:15.244843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.244813 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.245005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.244859 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.245145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.245127 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrspk\" (UniqueName: \"kubernetes.io/projected/5367ac19-8e88-4933-9ffc-52ae78419ac2-kube-api-access-xrspk\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.346409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.346376 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrspk\" (UniqueName: \"kubernetes.io/projected/5367ac19-8e88-4933-9ffc-52ae78419ac2-kube-api-access-xrspk\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.346590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.346458 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.346590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.346490 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.347274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.347248 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.348764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.348727 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.355027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.354999 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrspk\" (UniqueName: \"kubernetes.io/projected/5367ac19-8e88-4933-9ffc-52ae78419ac2-kube-api-access-xrspk\") pod \"maas-api-revoke-sub-eb3fe9dd-t9cws\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.359674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.359652 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:15.549686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.549614 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:15.840034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:15.840003 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws"] Nov 01 16:27:15.840851 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:27:15.840829 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5367ac19_8e88_4933_9ffc_52ae78419ac2.slice/crio-ef96722324bfa7ecd13840f2e5c632e11fb723940071475d375f19015cdc0a10 WatchSource:0}: Error finding container ef96722324bfa7ecd13840f2e5c632e11fb723940071475d375f19015cdc0a10: Status 404 returned error can't find the container with id ef96722324bfa7ecd13840f2e5c632e11fb723940071475d375f19015cdc0a10 Nov 01 16:27:16.067075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:16.067044 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:16.278014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:16.277985 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:16.702715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:16.702675 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" event={"ID":"5367ac19-8e88-4933-9ffc-52ae78419ac2","Type":"ContainerStarted","Data":"e48378aab9942188905c4ca7b84132f6358b3f133514329d0fc645a536ee35ac"} Nov 01 16:27:16.702952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:16.702743 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" event={"ID":"5367ac19-8e88-4933-9ffc-52ae78419ac2","Type":"ContainerStarted","Data":"ef96722324bfa7ecd13840f2e5c632e11fb723940071475d375f19015cdc0a10"} Nov 01 16:27:16.718567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:16.718516 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" podStartSLOduration=1.718498635 podStartE2EDuration="1.718498635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:16.716934661 +0000 UTC m=+2374.420100598" watchObservedRunningTime="2026-11-01 16:27:16.718498635 +0000 UTC m=+2374.421664572" Nov 01 16:27:16.976978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:16.976835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:17.682399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:17.682361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:17.712536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:17.712506 4573 generic.go:356] "Generic (PLEG): container finished" podID="5367ac19-8e88-4933-9ffc-52ae78419ac2" containerID="e48378aab9942188905c4ca7b84132f6358b3f133514329d0fc645a536ee35ac" exitCode=0 Nov 01 16:27:17.712765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:17.712552 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" event={"ID":"5367ac19-8e88-4933-9ffc-52ae78419ac2","Type":"ContainerDied","Data":"e48378aab9942188905c4ca7b84132f6358b3f133514329d0fc645a536ee35ac"} Nov 01 16:27:18.367991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:18.367959 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:18.918276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:18.918245 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:19.015277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.015178 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-xks-ca\") pod \"5367ac19-8e88-4933-9ffc-52ae78419ac2\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " Nov 01 16:27:19.015277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.015250 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5367ac19-8e88-4933-9ffc-52ae78419ac2-kube-api-access-xrspk\" (UniqueName: \"kubernetes.io/projected/5367ac19-8e88-4933-9ffc-52ae78419ac2-kube-api-access-xrspk\") pod \"5367ac19-8e88-4933-9ffc-52ae78419ac2\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " Nov 01 16:27:19.015536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.015372 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-openshift-ca\") pod \"5367ac19-8e88-4933-9ffc-52ae78419ac2\" (UID: \"5367ac19-8e88-4933-9ffc-52ae78419ac2\") " Nov 01 16:27:19.015943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.015912 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-openshift-ca" pod "5367ac19-8e88-4933-9ffc-52ae78419ac2" (UID: "5367ac19-8e88-4933-9ffc-52ae78419ac2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:19.016734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.016705 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:19.017612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.017589 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-xks-ca" pod "5367ac19-8e88-4933-9ffc-52ae78419ac2" (UID: "5367ac19-8e88-4933-9ffc-52ae78419ac2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:19.017709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.017631 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5367ac19-8e88-4933-9ffc-52ae78419ac2-kube-api-access-xrspk" pod "5367ac19-8e88-4933-9ffc-52ae78419ac2" (UID: "5367ac19-8e88-4933-9ffc-52ae78419ac2"). InnerVolumeSpecName "kube-api-access-xrspk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:19.066524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.066493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:19.140773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.123182 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5367ac19-8e88-4933-9ffc-52ae78419ac2-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:19.140773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.123233 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrspk\" (UniqueName: \"kubernetes.io/projected/5367ac19-8e88-4933-9ffc-52ae78419ac2-kube-api-access-xrspk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:19.731761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.731726 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" Nov 01 16:27:19.732172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.731780 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws" event={"ID":"5367ac19-8e88-4933-9ffc-52ae78419ac2","Type":"ContainerDied","Data":"ef96722324bfa7ecd13840f2e5c632e11fb723940071475d375f19015cdc0a10"} Nov 01 16:27:19.732172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.731843 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef96722324bfa7ecd13840f2e5c632e11fb723940071475d375f19015cdc0a10" Nov 01 16:27:19.790505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:19.790473 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:20.575943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:20.575912 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:21.070981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.070955 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq"] Nov 01 16:27:21.071792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.071759 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5367ac19-8e88-4933-9ffc-52ae78419ac2" containerName="revoke-keys" Nov 01 16:27:21.071792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.071778 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5367ac19-8e88-4933-9ffc-52ae78419ac2" containerName="revoke-keys" Nov 01 16:27:21.071956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.071922 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5367ac19-8e88-4933-9ffc-52ae78419ac2" containerName="revoke-keys" Nov 01 16:27:21.076084 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.076065 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.079231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.079209 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:27:21.080158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.080136 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:27:21.082220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.082198 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq"] Nov 01 16:27:21.156854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.156823 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.157059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.157034 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwb7p\" (UniqueName: \"kubernetes.io/projected/94a206da-61dd-410b-ba81-63aa82a1630a-kube-api-access-dwb7p\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.157107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.157076 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.258733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.258684 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.258943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.258912 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwb7p\" (UniqueName: \"kubernetes.io/projected/94a206da-61dd-410b-ba81-63aa82a1630a-kube-api-access-dwb7p\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.258990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.258948 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.259647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.259621 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.261092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.261069 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.266493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.266472 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwb7p\" (UniqueName: \"kubernetes.io/projected/94a206da-61dd-410b-ba81-63aa82a1630a-kube-api-access-dwb7p\") pod \"maas-api-revoke-sub-5c0c629a-pg9vq\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.394212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.394184 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:21.455581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.455235 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:21.678466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.678437 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq"] Nov 01 16:27:21.678640 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:27:21.678613 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a206da_61dd_410b_ba81_63aa82a1630a.slice/crio-ffc0d26e3fbbc61d1ee29fd808d7e21e42e336d769c9b557c2c5998287696521 WatchSource:0}: Error finding container ffc0d26e3fbbc61d1ee29fd808d7e21e42e336d769c9b557c2c5998287696521: Status 404 returned error can't find the container with id ffc0d26e3fbbc61d1ee29fd808d7e21e42e336d769c9b557c2c5998287696521 Nov 01 16:27:21.750279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:21.750248 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" event={"ID":"94a206da-61dd-410b-ba81-63aa82a1630a","Type":"ContainerStarted","Data":"ffc0d26e3fbbc61d1ee29fd808d7e21e42e336d769c9b557c2c5998287696521"} Nov 01 16:27:22.441599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:22.441557 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:22.762674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:22.762578 4573 generic.go:356] "Generic (PLEG): container finished" podID="94a206da-61dd-410b-ba81-63aa82a1630a" containerID="a2cc94efc3fdf0a83a25249f66e1b80c580c51bbb61e25620d53c9d0f82f0e73" exitCode=0 Nov 01 16:27:22.762872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:22.762659 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" event={"ID":"94a206da-61dd-410b-ba81-63aa82a1630a","Type":"ContainerDied","Data":"a2cc94efc3fdf0a83a25249f66e1b80c580c51bbb61e25620d53c9d0f82f0e73"} Nov 01 16:27:23.346322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:23.346279 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:23.943204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:23.943176 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:24.033678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.033592 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-openshift-ca\") pod \"94a206da-61dd-410b-ba81-63aa82a1630a\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " Nov 01 16:27:24.033904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.033732 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a206da-61dd-410b-ba81-63aa82a1630a-kube-api-access-dwb7p\" (UniqueName: \"kubernetes.io/projected/94a206da-61dd-410b-ba81-63aa82a1630a-kube-api-access-dwb7p\") pod \"94a206da-61dd-410b-ba81-63aa82a1630a\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " Nov 01 16:27:24.033992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.033902 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-xks-ca\") pod \"94a206da-61dd-410b-ba81-63aa82a1630a\" (UID: \"94a206da-61dd-410b-ba81-63aa82a1630a\") " Nov 01 16:27:24.034092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.034057 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-openshift-ca" pod "94a206da-61dd-410b-ba81-63aa82a1630a" (UID: "94a206da-61dd-410b-ba81-63aa82a1630a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:24.034504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.034478 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:24.036349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.036318 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a206da-61dd-410b-ba81-63aa82a1630a-kube-api-access-dwb7p" pod "94a206da-61dd-410b-ba81-63aa82a1630a" (UID: "94a206da-61dd-410b-ba81-63aa82a1630a"). InnerVolumeSpecName "kube-api-access-dwb7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:24.037826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.037803 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-xks-ca" pod "94a206da-61dd-410b-ba81-63aa82a1630a" (UID: "94a206da-61dd-410b-ba81-63aa82a1630a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:24.136253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.136220 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwb7p\" (UniqueName: \"kubernetes.io/projected/94a206da-61dd-410b-ba81-63aa82a1630a-kube-api-access-dwb7p\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:24.136253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.136250 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a206da-61dd-410b-ba81-63aa82a1630a-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:24.155655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.155619 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:24.366045 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:27:24.366006 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:27:24.784144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.784042 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" event={"ID":"94a206da-61dd-410b-ba81-63aa82a1630a","Type":"ContainerDied","Data":"ffc0d26e3fbbc61d1ee29fd808d7e21e42e336d769c9b557c2c5998287696521"} Nov 01 16:27:24.784144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.784113 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffc0d26e3fbbc61d1ee29fd808d7e21e42e336d769c9b557c2c5998287696521" Nov 01 16:27:24.784144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.784057 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq" Nov 01 16:27:24.866009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:24.865950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:25.576253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.576221 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t"] Nov 01 16:27:25.577165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.577145 4573 cpu_manager.go:414] "containerMap: removing container" podUID="94a206da-61dd-410b-ba81-63aa82a1630a" containerName="revoke-keys" Nov 01 16:27:25.577235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.577166 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a206da-61dd-410b-ba81-63aa82a1630a" containerName="revoke-keys" Nov 01 16:27:25.577339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.577330 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a206da-61dd-410b-ba81-63aa82a1630a" containerName="revoke-keys" Nov 01 16:27:25.581475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.581453 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.585569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.585537 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:27:25.585694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.585588 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:27:25.589662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.589637 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t"] Nov 01 16:27:25.660468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.660432 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.660620 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.660577 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.660682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.660659 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbftl\" (UniqueName: \"kubernetes.io/projected/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-kube-api-access-qbftl\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.762939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.762900 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.763135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.763060 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.763192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.763170 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbftl\" (UniqueName: \"kubernetes.io/projected/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-kube-api-access-qbftl\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.763730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.763706 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.765360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.765338 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.771162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.771138 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbftl\" (UniqueName: \"kubernetes.io/projected/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-kube-api-access-qbftl\") pod \"maas-api-revoke-sub-694a0e6a-mcm9t\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:25.780731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.780700 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:25.897856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:25.897822 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:26.185781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:26.185735 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t"] Nov 01 16:27:26.193013 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:27:26.192979 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76ad9212_eac0_44ea_a2af_0da2b25c2fd5.slice/crio-74f7503e514babe4c7d4d311ee31173c3c5234e4d6b022fcd717618931ec4ad8 WatchSource:0}: Error finding container 74f7503e514babe4c7d4d311ee31173c3c5234e4d6b022fcd717618931ec4ad8: Status 404 returned error can't find the container with id 74f7503e514babe4c7d4d311ee31173c3c5234e4d6b022fcd717618931ec4ad8 Nov 01 16:27:26.596246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:26.596210 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:26.803248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:26.803211 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" event={"ID":"76ad9212-eac0-44ea-a2af-0da2b25c2fd5","Type":"ContainerStarted","Data":"82343cfa3ef03cee231710c540e1bcd5063de9d7b93899129878974a79da5c77"} Nov 01 16:27:26.803425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:26.803268 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" event={"ID":"76ad9212-eac0-44ea-a2af-0da2b25c2fd5","Type":"ContainerStarted","Data":"74f7503e514babe4c7d4d311ee31173c3c5234e4d6b022fcd717618931ec4ad8"} Nov 01 16:27:26.820704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:26.820595 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" podStartSLOduration=1.82057988 podStartE2EDuration="1.82057988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:26.818538373 +0000 UTC m=+2384.521704309" watchObservedRunningTime="2026-11-01 16:27:26.82057988 +0000 UTC m=+2384.523745847" Nov 01 16:27:27.235040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.235010 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg"] Nov 01 16:27:27.240896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.240869 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.245062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.245038 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7321d6-serving-cert\"" Nov 01 16:27:27.251214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.251190 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg"] Nov 01 16:27:27.293035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.292998 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-metrics-certs\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.293223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.293185 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a08917-2376-41da-b0bb-f814b4790e32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.293405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.293379 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zb9t\" (UniqueName: \"kubernetes.io/projected/c7a08917-2376-41da-b0bb-f814b4790e32-kube-api-access-5zb9t\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.293569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.293536 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.338720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.338663 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:27.438004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.437913 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zb9t\" (UniqueName: \"kubernetes.io/projected/c7a08917-2376-41da-b0bb-f814b4790e32-kube-api-access-5zb9t\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.438858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.438823 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.443920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.443894 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-metrics-certs\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.456021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.444256 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a08917-2376-41da-b0bb-f814b4790e32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.457255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.449624 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-metrics-certs\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.457255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.442110 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:27.457909 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:27:27.442447 4573 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-serving-cert: secret "maas-api-e2e-ipp-b-7321d6-serving-cert" not found Nov 01 16:27:27.457909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.454630 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zb9t\" (UniqueName: \"kubernetes.io/projected/c7a08917-2376-41da-b0bb-f814b4790e32-kube-api-access-5zb9t\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.460370 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:27:27.459720 4573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls podName:c7a08917-2376-41da-b0bb-f814b4790e32 nodeName:}" failed. No retries permitted until 2026-11-01 16:27:27.959680038 +0000 UTC m=+2385.662845964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls") pod "maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" (UID: "c7a08917-2376-41da-b0bb-f814b4790e32") : secret "maas-api-e2e-ipp-b-7321d6-serving-cert" not found Nov 01 16:27:27.489644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.489587 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a08917-2376-41da-b0bb-f814b4790e32-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.591995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.591958 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:27.813227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.813132 4573 generic.go:356] "Generic (PLEG): container finished" podID="76ad9212-eac0-44ea-a2af-0da2b25c2fd5" containerID="82343cfa3ef03cee231710c540e1bcd5063de9d7b93899129878974a79da5c77" exitCode=0 Nov 01 16:27:27.813627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.813237 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" event={"ID":"76ad9212-eac0-44ea-a2af-0da2b25c2fd5","Type":"ContainerDied","Data":"82343cfa3ef03cee231710c540e1bcd5063de9d7b93899129878974a79da5c77"} Nov 01 16:27:27.971775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.971463 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:27.974251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:27.974225 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:28.159856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:28.159817 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:28.259597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:28.259563 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:28.445005 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:27:28.444976 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a08917_2376_41da_b0bb_f814b4790e32.slice/crio-827e60c602db2283ac3eca61a2ad52fd457830388081cf3d630744b7ecd753e7 WatchSource:0}: Error finding container 827e60c602db2283ac3eca61a2ad52fd457830388081cf3d630744b7ecd753e7: Status 404 returned error can't find the container with id 827e60c602db2283ac3eca61a2ad52fd457830388081cf3d630744b7ecd753e7 Nov 01 16:27:28.445554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:28.445530 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg"] Nov 01 16:27:28.825818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:28.825691 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" event={"ID":"c7a08917-2376-41da-b0bb-f814b4790e32","Type":"ContainerStarted","Data":"827e60c602db2283ac3eca61a2ad52fd457830388081cf3d630744b7ecd753e7"} Nov 01 16:27:29.161776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.161109 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:29.197087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.197060 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-openshift-ca\") pod \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " Nov 01 16:27:29.197231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.197216 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-xks-ca\") pod \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " Nov 01 16:27:29.197290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.197278 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-kube-api-access-qbftl\" (UniqueName: \"kubernetes.io/projected/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-kube-api-access-qbftl\") pod \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\" (UID: \"76ad9212-eac0-44ea-a2af-0da2b25c2fd5\") " Nov 01 16:27:29.197522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.197496 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-openshift-ca" pod "76ad9212-eac0-44ea-a2af-0da2b25c2fd5" (UID: "76ad9212-eac0-44ea-a2af-0da2b25c2fd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:29.198096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.197952 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:29.200658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.200618 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-xks-ca" pod "76ad9212-eac0-44ea-a2af-0da2b25c2fd5" (UID: "76ad9212-eac0-44ea-a2af-0da2b25c2fd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:29.200862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.200839 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-kube-api-access-qbftl" pod "76ad9212-eac0-44ea-a2af-0da2b25c2fd5" (UID: "76ad9212-eac0-44ea-a2af-0da2b25c2fd5"). InnerVolumeSpecName "kube-api-access-qbftl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:29.299908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.299872 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:29.299908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.299904 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbftl\" (UniqueName: \"kubernetes.io/projected/76ad9212-eac0-44ea-a2af-0da2b25c2fd5-kube-api-access-qbftl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:29.440848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.440774 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:29.835728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.835630 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" event={"ID":"c7a08917-2376-41da-b0bb-f814b4790e32","Type":"ContainerStarted","Data":"315e0306e2020f0ecf028c9c08e98f0e018ed1080d68482fdd53f0ae9559d313"} Nov 01 16:27:29.836269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.835760 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:29.837170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.837151 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" Nov 01 16:27:29.837307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.837216 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t" event={"ID":"76ad9212-eac0-44ea-a2af-0da2b25c2fd5","Type":"ContainerDied","Data":"74f7503e514babe4c7d4d311ee31173c3c5234e4d6b022fcd717618931ec4ad8"} Nov 01 16:27:29.837307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.837250 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74f7503e514babe4c7d4d311ee31173c3c5234e4d6b022fcd717618931ec4ad8" Nov 01 16:27:29.852148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:29.852107 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" podStartSLOduration=2.350259186 podStartE2EDuration="2.852095594s" totalImagesPullingTime="501.836408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:29.851498585 +0000 UTC m=+2387.554664523" watchObservedRunningTime="2026-11-01 16:27:29.852095594 +0000 UTC m=+2387.555261530" Nov 01 16:27:30.256533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:30.256493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:30.894341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:30.894310 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:31.833449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:31.833416 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:32.177098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:32.177054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:32.549640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:32.549527 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:32.955284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:32.955247 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s"] Nov 01 16:27:32.956069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:32.956054 4573 cpu_manager.go:414] "containerMap: removing container" podUID="76ad9212-eac0-44ea-a2af-0da2b25c2fd5" containerName="revoke-keys" Nov 01 16:27:32.956124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:32.956070 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ad9212-eac0-44ea-a2af-0da2b25c2fd5" containerName="revoke-keys" Nov 01 16:27:32.956194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:32.956184 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ad9212-eac0-44ea-a2af-0da2b25c2fd5" containerName="revoke-keys" Nov 01 16:27:32.960390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:32.960367 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:32.967645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:32.967615 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s"] Nov 01 16:27:33.075705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.075671 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5gm\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-kube-api-access-4s5gm\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.075912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.075726 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.075912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.075832 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dadf08-84bd-445e-85d7-f2795f22b4c1-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.075912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.075898 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.076089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.075934 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.076089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.076043 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.076201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.076102 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.076201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.076192 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.076311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.076293 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.177981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.177938 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5gm\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-kube-api-access-4s5gm\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178005 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178064 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dadf08-84bd-445e-85d7-f2795f22b4c1-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178143 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178194 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178266 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178325 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178419 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178425 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178550 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178624 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178809 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178925 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.178970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.178963 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dadf08-84bd-445e-85d7-f2795f22b4c1-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.180663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.180639 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.181140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.181119 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.186073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.186054 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5gm\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-kube-api-access-4s5gm\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.186165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.186097 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.259656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.259550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:33.277501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.277474 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:33.602424 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:27:33.602396 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31dadf08_84bd_445e_85d7_f2795f22b4c1.slice/crio-21b9d456b42132c33973094852976eae0a94c830e4c73f4890947343a8f666e1 WatchSource:0}: Error finding container 21b9d456b42132c33973094852976eae0a94c830e4c73f4890947343a8f666e1: Status 404 returned error can't find the container with id 21b9d456b42132c33973094852976eae0a94c830e4c73f4890947343a8f666e1 Nov 01 16:27:33.603592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.603566 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s"] Nov 01 16:27:33.605897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.605868 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:33.606016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.605930 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:33.606016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.605956 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:27:33.875243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.875207 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" event={"ID":"31dadf08-84bd-445e-85d7-f2795f22b4c1","Type":"ContainerStarted","Data":"fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db"} Nov 01 16:27:33.875452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.875263 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" event={"ID":"31dadf08-84bd-445e-85d7-f2795f22b4c1","Type":"ContainerStarted","Data":"21b9d456b42132c33973094852976eae0a94c830e4c73f4890947343a8f666e1"} Nov 01 16:27:33.904679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.903247 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" podStartSLOduration=1.903229959 podStartE2EDuration="1.903229959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:27:33.892694134 +0000 UTC m=+2391.595860071" watchObservedRunningTime="2026-11-01 16:27:33.903229959 +0000 UTC m=+2391.606395895" Nov 01 16:27:33.958017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:33.957975 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:34.278263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:34.278186 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:34.279960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:34.279934 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 16:27:34.280080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:34.279979 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 16:27:34.788618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:34.788582 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:35.281269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:35.281224 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 16:27:35.282076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:35.282039 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 16:27:35.684774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:35.683294 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:35.851826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:35.851793 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:27:36.282774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:36.281061 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 16:27:36.282774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:36.281124 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 16:27:36.427029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:36.426993 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:37.286776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:37.285851 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 16:27:37.286776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:37.285914 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 16:27:37.414936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:37.414903 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:37.539937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:37.539830 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:38.286878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:38.286840 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 16:27:38.287400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:38.286898 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 16:27:38.512938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:38.512899 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:39.281791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:39.281745 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" start-of-body= Nov 01 16:27:39.282126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:39.282075 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.9:15021/healthz/ready\": dial tcp 10.133.1.9:15021: connect: connection refused" Nov 01 16:27:39.385820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:39.385783 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:40.185419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:40.185384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:40.306948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:40.306912 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:40.307556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:40.307518 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:40.308287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:40.308264 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:27:40.971019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:40.970986 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:42.123051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:42.123005 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:42.743098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:42.743066 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:43.301608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:43.301583 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:27:43.312737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:43.312712 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:27:43.314047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:43.314026 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:27:43.324220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:43.324199 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:27:43.465450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:43.465421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:44.201578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:44.201541 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:45.046182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:45.046145 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:45.964539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:45.964498 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:46.357410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:46.357382 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn"] Nov 01 16:27:46.357908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:46.357622 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" containerID="cri-o://2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa" gracePeriod=30 Nov 01 16:27:46.726973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:46.726882 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:47.464602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:47.464567 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:48.403099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:48.403060 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:48.761657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:48.761584 4573 scope.go:122] "RemoveContainer" containerID="0e9dcf30619159407c9bb625e0432cc8c7774a4aedb5679b5db70d6f5b8931d6" Nov 01 16:27:48.773179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:48.773161 4573 scope.go:122] "RemoveContainer" containerID="8117dfb52051936dcda31922aacc739b0ff3371e005d6c7d0101517d9cea8286" Nov 01 16:27:48.783282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:48.783260 4573 scope.go:122] "RemoveContainer" containerID="6813356da2972d9b618f4589b6a2cbc3278200a6331a8f2ba795d60b375f19eb" Nov 01 16:27:48.792605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:48.792585 4573 scope.go:122] "RemoveContainer" containerID="cb535016728b200ede6fde54538e3c4cc5a20cf19d575532fa5450cd3591d5e8" Nov 01 16:27:49.110259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:49.110223 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:49.211977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:49.211936 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:49.323248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:49.323205 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:49.566830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:49.566477 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:50.457356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:50.457316 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:51.170251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.170219 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:51.662162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.662136 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:27:51.771901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.771826 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-certs\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.771912 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-token\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-token\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.771968 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-podinfo\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772002 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-envoy\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772084 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-credential-socket\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772113 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-certs" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:51.772350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772260 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-socket\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772297 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istiod-ca-cert\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772345 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-kube-api-access-69cct\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-kube-api-access-69cct\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772409 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-credential-socket" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:51.772523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772438 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-data\") pod \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\" (UID: \"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b\") " Nov 01 16:27:51.772652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772537 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-socket" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:51.772795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772767 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istiod-ca-cert" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:51.772795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.772742 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-data" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:51.773238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.773213 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:51.773295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.773247 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:51.773295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.773262 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:51.773295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.773276 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:51.773295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.773289 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:51.774377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.774339 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-envoy" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:27:51.774672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.774652 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-podinfo" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:27:51.774915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.774897 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-kube-api-access-69cct" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "kube-api-access-69cct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:51.775110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.775090 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-token" pod "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" (UID: "e763fd9f-5b6f-4086-83c3-8b5f4a0de00b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:51.875029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.874995 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69cct\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-kube-api-access-69cct\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:51.875029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.875025 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:51.875029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.875036 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:51.875275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:51.875046 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:52.043251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.043168 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:52.142954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.142909 4573 generic.go:356] "Generic (PLEG): container finished" podID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerID="2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa" exitCode=0 Nov 01 16:27:52.153173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.143246 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" Nov 01 16:27:52.153173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.143265 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" event={"ID":"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b","Type":"ContainerDied","Data":"2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa"} Nov 01 16:27:52.153173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.148502 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn" event={"ID":"e763fd9f-5b6f-4086-83c3-8b5f4a0de00b","Type":"ContainerDied","Data":"a821be868061cf361891a4d418f276f73c1b2643794f07210abbe2c22019cdfd"} Nov 01 16:27:52.153173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.148546 4573 scope.go:122] "RemoveContainer" containerID="2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa" Nov 01 16:27:52.169820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.169801 4573 scope.go:122] "RemoveContainer" containerID="2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa" Nov 01 16:27:52.170130 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:27:52.170112 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa\": container with ID starting with 2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa not found: ID does not exist" containerID="2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa" Nov 01 16:27:52.170178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.170140 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa"} err="failed to get container status \"2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa\": rpc error: code = NotFound desc = could not find container \"2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa\": container with ID starting with 2b6e00a2453bbf2a71e44d336aa83968242935ff7ae5384400f718c596045dfa not found: ID does not exist" Nov 01 16:27:52.178044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.178018 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn"] Nov 01 16:27:52.183476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.183453 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-zrljn"] Nov 01 16:27:52.431858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.431822 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t"] Nov 01 16:27:52.432587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.432573 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" Nov 01 16:27:52.432633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.432588 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" Nov 01 16:27:52.432733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.432724 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" containerName="istio-proxy" Nov 01 16:27:52.438436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.438412 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.441781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.441613 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t"] Nov 01 16:27:52.441781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.441639 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:27:52.441781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.441709 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:27:52.466495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.466458 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e763fd9f-5b6f-4086-83c3-8b5f4a0de00b" path="/var/lib/kubelet/pods/e763fd9f-5b6f-4086-83c3-8b5f4a0de00b/volumes" Nov 01 16:27:52.482000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.481967 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.482178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.482054 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfpnw\" (UniqueName: \"kubernetes.io/projected/65901982-a0f2-4b25-9efb-22f85a8eb245-kube-api-access-jfpnw\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.482350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.482325 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.583969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.583936 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.583969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.583975 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpnw\" (UniqueName: \"kubernetes.io/projected/65901982-a0f2-4b25-9efb-22f85a8eb245-kube-api-access-jfpnw\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.584247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.584142 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.584787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.584745 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.586794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.586730 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.591717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.591697 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpnw\" (UniqueName: \"kubernetes.io/projected/65901982-a0f2-4b25-9efb-22f85a8eb245-kube-api-access-jfpnw\") pod \"maas-api-revoke-sub-a9ad2c91-bw75t\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.761062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.760981 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:52.769924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:52.769898 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:53.046359 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:27:53.046315 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65901982_a0f2_4b25_9efb_22f85a8eb245.slice/crio-cdcf246f9848f2ed88c97c78f8b736e21ac4a36a6efba7657b2382a6ed835683 WatchSource:0}: Error finding container cdcf246f9848f2ed88c97c78f8b736e21ac4a36a6efba7657b2382a6ed835683: Status 404 returned error can't find the container with id cdcf246f9848f2ed88c97c78f8b736e21ac4a36a6efba7657b2382a6ed835683 Nov 01 16:27:53.046560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:53.046538 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t"] Nov 01 16:27:53.158483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:53.158451 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" event={"ID":"65901982-a0f2-4b25-9efb-22f85a8eb245","Type":"ContainerStarted","Data":"cdcf246f9848f2ed88c97c78f8b736e21ac4a36a6efba7657b2382a6ed835683"} Nov 01 16:27:53.486136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:53.486108 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:54.161786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:54.161736 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:54.170190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:54.170165 4573 generic.go:356] "Generic (PLEG): container finished" podID="65901982-a0f2-4b25-9efb-22f85a8eb245" containerID="34b30a3782e647950f1635882abc5ea8375159cfada7f348c35fec62184a10ed" exitCode=0 Nov 01 16:27:54.170340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:54.170253 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" event={"ID":"65901982-a0f2-4b25-9efb-22f85a8eb245","Type":"ContainerDied","Data":"34b30a3782e647950f1635882abc5ea8375159cfada7f348c35fec62184a10ed"} Nov 01 16:27:54.367414 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:27:54.367378 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:27:54.925440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:54.925405 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:55.352677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.352652 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:55.468006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.467906 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-openshift-ca\") pod \"65901982-a0f2-4b25-9efb-22f85a8eb245\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " Nov 01 16:27:55.468006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.467979 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65901982-a0f2-4b25-9efb-22f85a8eb245-kube-api-access-jfpnw\" (UniqueName: \"kubernetes.io/projected/65901982-a0f2-4b25-9efb-22f85a8eb245-kube-api-access-jfpnw\") pod \"65901982-a0f2-4b25-9efb-22f85a8eb245\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " Nov 01 16:27:55.468268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.468137 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-xks-ca\") pod \"65901982-a0f2-4b25-9efb-22f85a8eb245\" (UID: \"65901982-a0f2-4b25-9efb-22f85a8eb245\") " Nov 01 16:27:55.468268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.468251 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-openshift-ca" pod "65901982-a0f2-4b25-9efb-22f85a8eb245" (UID: "65901982-a0f2-4b25-9efb-22f85a8eb245"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:27:55.468825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.468797 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:55.470163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.470137 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-xks-ca" pod "65901982-a0f2-4b25-9efb-22f85a8eb245" (UID: "65901982-a0f2-4b25-9efb-22f85a8eb245"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:27:55.470267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.470171 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65901982-a0f2-4b25-9efb-22f85a8eb245-kube-api-access-jfpnw" pod "65901982-a0f2-4b25-9efb-22f85a8eb245" (UID: "65901982-a0f2-4b25-9efb-22f85a8eb245"). InnerVolumeSpecName "kube-api-access-jfpnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:27:55.570652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.570565 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfpnw\" (UniqueName: \"kubernetes.io/projected/65901982-a0f2-4b25-9efb-22f85a8eb245-kube-api-access-jfpnw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:55.570652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.570599 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65901982-a0f2-4b25-9efb-22f85a8eb245-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:27:55.767354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:55.767239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:56.187461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:56.187424 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" Nov 01 16:27:56.187623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:56.187458 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t" event={"ID":"65901982-a0f2-4b25-9efb-22f85a8eb245","Type":"ContainerDied","Data":"cdcf246f9848f2ed88c97c78f8b736e21ac4a36a6efba7657b2382a6ed835683"} Nov 01 16:27:56.187623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:56.187503 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdcf246f9848f2ed88c97c78f8b736e21ac4a36a6efba7657b2382a6ed835683" Nov 01 16:27:56.464816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:56.464700 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:57.218799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:57.218765 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:57.949579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:57.949533 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:58.647434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:58.647382 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:59.279788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:59.279722 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:27:59.909436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:27:59.909402 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:00.009998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:00.009953 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:00.112311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:00.112265 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:00.747995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:00.747956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:01.117379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:01.117350 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w"] Nov 01 16:28:01.117610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:01.117584 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" containerID="cri-o://41fed38f636e146318bca59309a081783923e74dabb273585798877717098fe8" gracePeriod=30 Nov 01 16:28:01.471280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:01.471206 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:01.749318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:01.749234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:01.849368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:01.849334 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:02.263491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:02.263455 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:02.968984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:02.968940 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:03.078617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:03.078583 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:03.855684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:03.855652 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:04.854544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:04.854504 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:05.681113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:05.681083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:05.794197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:05.794164 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:28:06.289796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.289732 4573 generic.go:356] "Generic (PLEG): container finished" podID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerID="41fed38f636e146318bca59309a081783923e74dabb273585798877717098fe8" exitCode=0 Nov 01 16:28:06.290269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.289812 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" event={"ID":"9e3909dc-41fc-4299-ad14-5a36d69318d8","Type":"ContainerDied","Data":"41fed38f636e146318bca59309a081783923e74dabb273585798877717098fe8"} Nov 01 16:28:06.439335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.439309 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:28:06.489035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.488998 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-credential-socket\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489067 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-certs\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489131 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-podinfo\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489215 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-socket\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489260 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-token\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-token\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489341 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-envoy\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489352 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-credential-socket" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:06.489603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489439 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-kube-api-access-2kppw\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-kube-api-access-2kppw\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489476 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3909dc-41fc-4299-ad14-5a36d69318d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3909dc-41fc-4299-ad14-5a36d69318d8-istiod-ca-cert\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489607 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-data\") pod \"9e3909dc-41fc-4299-ad14-5a36d69318d8\" (UID: \"9e3909dc-41fc-4299-ad14-5a36d69318d8\") " Nov 01 16:28:06.489828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489769 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-certs" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:06.489828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.489805 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-socket" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:06.490676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.490195 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3909dc-41fc-4299-ad14-5a36d69318d8-istiod-ca-cert" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:06.490676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.490379 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e3909dc-41fc-4299-ad14-5a36d69318d8-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.490676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.490405 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.490676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.490421 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.490676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.490436 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.490676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.490641 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-data" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:06.492129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.492100 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-podinfo" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:06.492247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.492148 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-token" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:06.492651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.492626 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-envoy" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:06.493595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.493573 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-kube-api-access-2kppw" pod "9e3909dc-41fc-4299-ad14-5a36d69318d8" (UID: "9e3909dc-41fc-4299-ad14-5a36d69318d8"). InnerVolumeSpecName "kube-api-access-2kppw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:06.591570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.591538 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kppw\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-kube-api-access-2kppw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.591570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.591569 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.591570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.591580 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.591862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.591589 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.591862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.591599 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e3909dc-41fc-4299-ad14-5a36d69318d8-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:06.883169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:06.883139 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:07.371978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:07.371942 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" event={"ID":"9e3909dc-41fc-4299-ad14-5a36d69318d8","Type":"ContainerDied","Data":"22f3488eaac43935680f73dc4ec237386e2a74198ba3b9fe1435accb2dd5bdf3"} Nov 01 16:28:07.372587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:07.372565 4573 scope.go:122] "RemoveContainer" containerID="41fed38f636e146318bca59309a081783923e74dabb273585798877717098fe8" Nov 01 16:28:07.372956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:07.372937 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w" Nov 01 16:28:07.506783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:07.506737 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w"] Nov 01 16:28:07.513344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:07.513311 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-43fac8-openshift-default-84b4466d8c-t799w"] Nov 01 16:28:07.799187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:07.799083 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:08.290775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.288105 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" podUID="38865718-c2c2-48c3-aa91-47ce3a4fb022" containerName="main" probeResult="failure" output="Get \"https://10.133.0.235:8000/ready\": context deadline exceeded" Nov 01 16:28:08.398894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.398829 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" podUID="5dd17200-6b81-4260-9240-9615be94a2f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.236:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.475776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.403837 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.475776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.431853 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.475776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.453103 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded" Nov 01 16:28:08.740775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.733317 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.740775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.739558 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.765810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.744543 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt" podUID="a0218c49-8d6d-4bb3-8cf7-10634e6c7a44" containerName="main" probeResult="failure" output="Get \"https://10.133.0.33:8000/ready\": context deadline exceeded" Nov 01 16:28:08.801776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.801507 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 01 16:28:08.854774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.830522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:08.882775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.869152 4573 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 16:28:08.925900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:08.925840 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": context deadline exceeded" Nov 01 16:28:09.130004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:09.129964 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" path="/var/lib/kubelet/pods/9e3909dc-41fc-4299-ad14-5a36d69318d8/volumes" Nov 01 16:28:09.137775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:09.130459 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f"] Nov 01 16:28:09.137775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:09.130508 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-48e07fc0-rww9f"] Nov 01 16:28:09.763051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:09.763014 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:10.460599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:10.460565 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa04960-f591-4f15-8c63-07e571e77dbb" path="/var/lib/kubelet/pods/daa04960-f591-4f15-8c63-07e571e77dbb/volumes" Nov 01 16:28:10.883056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:10.883022 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:11.793568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:11.793535 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:12.160159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:12.160124 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:12.773398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:12.773361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:13.580051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:13.580017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:13.845512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:13.845392 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:14.203435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:14.203404 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:14.293513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:14.293477 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:14.658880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:14.658851 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:15.058861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:15.058714 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:15.854153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:15.854119 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:16.663817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:16.663765 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:17.393726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:17.393691 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:18.087824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:18.087790 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:18.846304 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:18.846239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:19.551680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:19.551641 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:19.847116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:19.847017 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:20.362477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:20.362447 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:21.084320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:21.084286 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:21.211943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:21.211909 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:21.384599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:21.384558 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c"] Nov 01 16:28:21.385098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:21.385059 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" containerID="cri-o://2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7" gracePeriod=30 Nov 01 16:28:21.416417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:21.416383 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:22.287511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:22.287468 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:23.024867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:23.024833 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59"] Nov 01 16:28:23.032058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:23.032031 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a6e3b6f-sst59"] Nov 01 16:28:23.159083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:23.159052 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:23.970256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:23.970217 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:24.366037 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:24.366004 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:28:24.461276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:24.461247 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476c00ef-8bd0-4f86-8f66-a13182d309f2" path="/var/lib/kubelet/pods/476c00ef-8bd0-4f86-8f66-a13182d309f2/volumes" Nov 01 16:28:24.783725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:24.783626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:25.128973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:25.128937 4573 patch_prober.go:29] interesting pod/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:28:25.129493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:25.128995 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:28:25.718179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:25.718145 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:26.487645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.487607 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:26.737805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.737781 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:28:26.899067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899029 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-data\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899143 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-credential-socket\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899197 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-istio-token\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-istio-token\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899281 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-kube-api-access-pk6h9\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-kube-api-access-pk6h9\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899343 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-data" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.899395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899369 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-credential-socket" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.899395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899377 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-envoy\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899451 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-socket\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899533 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a3149b1-0513-478e-8c0f-633766afc158-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3149b1-0513-478e-8c0f-633766afc158-istio-podinfo\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899562 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3149b1-0513-478e-8c0f-633766afc158-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3149b1-0513-478e-8c0f-633766afc158-istiod-ca-cert\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899663 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-certs\") pod \"8a3149b1-0513-478e-8c0f-633766afc158\" (UID: \"8a3149b1-0513-478e-8c0f-633766afc158\") " Nov 01 16:28:26.899884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.899820 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-socket" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.901045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.900430 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.901045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.900460 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.901045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.900477 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:26.901045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.901008 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3149b1-0513-478e-8c0f-633766afc158-istiod-ca-cert" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:26.901509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.901472 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-certs" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:26.901861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.901836 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-istio-token" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:26.902187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.902165 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-kube-api-access-pk6h9" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "kube-api-access-pk6h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:26.902274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.902203 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a3149b1-0513-478e-8c0f-633766afc158-istio-podinfo" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:26.902274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:26.902220 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-envoy" pod "8a3149b1-0513-478e-8c0f-633766afc158" (UID: "8a3149b1-0513-478e-8c0f-633766afc158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:27.002664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.002584 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.002664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.002621 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk6h9\" (UniqueName: \"kubernetes.io/projected/8a3149b1-0513-478e-8c0f-633766afc158-kube-api-access-pk6h9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.002664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.002638 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.002664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.002654 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a3149b1-0513-478e-8c0f-633766afc158-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.003040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.002668 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a3149b1-0513-478e-8c0f-633766afc158-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.003040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.002683 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a3149b1-0513-478e-8c0f-633766afc158-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:27.291988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.291916 4573 generic.go:356] "Generic (PLEG): container finished" podID="8a3149b1-0513-478e-8c0f-633766afc158" containerID="2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7" exitCode=0 Nov 01 16:28:27.292147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.292023 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" Nov 01 16:28:27.292147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.292054 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" event={"ID":"8a3149b1-0513-478e-8c0f-633766afc158","Type":"ContainerDied","Data":"2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7"} Nov 01 16:28:27.292147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.292114 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c" event={"ID":"8a3149b1-0513-478e-8c0f-633766afc158","Type":"ContainerDied","Data":"fb3b979e38e241d5e4f14f4ae79f6666d74fecde0025a11ab92e82127901b67e"} Nov 01 16:28:27.292147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.292144 4573 scope.go:122] "RemoveContainer" containerID="2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7" Nov 01 16:28:27.304477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.304458 4573 scope.go:122] "RemoveContainer" containerID="2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7" Nov 01 16:28:27.304775 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:27.304732 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7\": container with ID starting with 2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7 not found: ID does not exist" containerID="2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7" Nov 01 16:28:27.304882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.304773 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7"} err="failed to get container status \"2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7\": rpc error: code = NotFound desc = could not find container \"2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7\": container with ID starting with 2f951c05863e30fa18359751db93f54885fc5ba2d2aac83fd018d97b1a549ec7 not found: ID does not exist" Nov 01 16:28:27.319955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.319934 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c"] Nov 01 16:28:27.324461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.324431 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-7225e5-openshift-default-59ccbb9cdc-q495c"] Nov 01 16:28:27.613431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:27.613395 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:28.406547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:28.406516 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:28.460825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:28.460794 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3149b1-0513-478e-8c0f-633766afc158" path="/var/lib/kubelet/pods/8a3149b1-0513-478e-8c0f-633766afc158/volumes" Nov 01 16:28:29.111191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:29.111160 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:30.164407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:30.164378 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:30.938683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:30.938648 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:31.880455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:31.880403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:32.065325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:32.065291 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt"] Nov 01 16:28:32.097525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:32.097494 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb"] Nov 01 16:28:32.131247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:32.130697 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:32.200713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:32.200679 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:32.301636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:32.301598 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:32.875053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:32.875018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:33.063049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.062773 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x"] Nov 01 16:28:33.064206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064181 4573 cpu_manager.go:414] "containerMap: removing container" podUID="65901982-a0f2-4b25-9efb-22f85a8eb245" containerName="revoke-keys" Nov 01 16:28:33.064389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064365 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65901982-a0f2-4b25-9efb-22f85a8eb245" containerName="revoke-keys" Nov 01 16:28:33.064514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064409 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" Nov 01 16:28:33.064514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064421 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" Nov 01 16:28:33.064514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064449 4573 cpu_manager.go:414] "containerMap: removing container" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" Nov 01 16:28:33.064514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064456 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" Nov 01 16:28:33.064740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064714 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="65901982-a0f2-4b25-9efb-22f85a8eb245" containerName="revoke-keys" Nov 01 16:28:33.064740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064739 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3149b1-0513-478e-8c0f-633766afc158" containerName="istio-proxy" Nov 01 16:28:33.064884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.064767 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3909dc-41fc-4299-ad14-5a36d69318d8" containerName="istio-proxy" Nov 01 16:28:33.070500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.070471 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.079971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.078559 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x"] Nov 01 16:28:33.115721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.115687 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.115903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.115745 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.115950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.115891 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.115950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.115944 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe880f4e-50c1-40db-9900-87ac858f8e9d-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.116063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.116051 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.116101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.116077 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.116169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.116152 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.116210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.116188 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.116278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.116266 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp69p\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-kube-api-access-fp69p\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217073 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217134 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217160 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217194 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp69p\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-kube-api-access-fp69p\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217295 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217374 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217431 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217481 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe880f4e-50c1-40db-9900-87ac858f8e9d-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217500 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-credential-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217572 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217590 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-certs\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.217925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217896 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-data\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.218067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.217994 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-socket\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.218219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.218199 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe880f4e-50c1-40db-9900-87ac858f8e9d-istiod-ca-cert\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.219592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.219569 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-envoy\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.219920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.219901 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-podinfo\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.227670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.227647 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-token\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.230324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.230300 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp69p\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-kube-api-access-fp69p\") pod \"e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.399610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.399577 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:33.596032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.596004 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr"] Nov 01 16:28:33.602081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.602060 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.610066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.610040 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-a9a967-openshift-default-dockercfg-8bwnt\"" Nov 01 16:28:33.614499 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.614476 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr"] Nov 01 16:28:33.688091 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:28:33.688052 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe880f4e_50c1_40db_9900_87ac858f8e9d.slice/crio-2e6fd629f7a33ffa8d778a35baabdaaf05432859ca05b145855ee61afe8a2b54 WatchSource:0}: Error finding container 2e6fd629f7a33ffa8d778a35baabdaaf05432859ca05b145855ee61afe8a2b54: Status 404 returned error can't find the container with id 2e6fd629f7a33ffa8d778a35baabdaaf05432859ca05b145855ee61afe8a2b54 Nov 01 16:28:33.689783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.689735 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x"] Nov 01 16:28:33.691687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.691654 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:33.691776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.691718 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:33.691822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.691785 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:33.726122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726098 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7n9s\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-kube-api-access-z7n9s\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.726234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726150 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.726356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726323 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.726415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726401 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.726543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726518 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.726642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726624 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.726691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726670 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b84f57-6d13-4519-8167-50bafbbcedfe-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.726811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726796 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.726915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.726902 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.828407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828361 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.828657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828470 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.828657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828517 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b84f57-6d13-4519-8167-50bafbbcedfe-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.828657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828587 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.828914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828723 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.828914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828783 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7n9s\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-kube-api-access-z7n9s\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.828914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828827 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.828914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828911 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.829902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.828980 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.829902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.829046 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.829902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.829130 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.829902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.829471 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.829902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.829677 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.830104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.830017 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b84f57-6d13-4519-8167-50bafbbcedfe-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.831678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.831654 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.832232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.832204 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.838266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.838243 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.838542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.838520 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7n9s\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-kube-api-access-z7n9s\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:33.853801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.853743 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:33.947208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:33.947176 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:34.243209 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:28:34.243175 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b84f57_6d13_4519_8167_50bafbbcedfe.slice/crio-41eb3d29cc757dd43678d809000541589c0c32cf9956972b07195964b9a110db WatchSource:0}: Error finding container 41eb3d29cc757dd43678d809000541589c0c32cf9956972b07195964b9a110db: Status 404 returned error can't find the container with id 41eb3d29cc757dd43678d809000541589c0c32cf9956972b07195964b9a110db Nov 01 16:28:34.243690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.243667 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr"] Nov 01 16:28:34.246835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.246806 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:34.246924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.246873 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:34.246924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.246900 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:28:34.410926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.410886 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" event={"ID":"fe880f4e-50c1-40db-9900-87ac858f8e9d","Type":"ContainerStarted","Data":"46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264"} Nov 01 16:28:34.411097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.410953 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" event={"ID":"fe880f4e-50c1-40db-9900-87ac858f8e9d","Type":"ContainerStarted","Data":"2e6fd629f7a33ffa8d778a35baabdaaf05432859ca05b145855ee61afe8a2b54"} Nov 01 16:28:34.414025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.413994 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" event={"ID":"54b84f57-6d13-4519-8167-50bafbbcedfe","Type":"ContainerStarted","Data":"bfaa74f984a58884a2c3789f9708635fd6a1d0e6b3197dbea6b831c8e83ecb28"} Nov 01 16:28:34.414166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.414044 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" event={"ID":"54b84f57-6d13-4519-8167-50bafbbcedfe","Type":"ContainerStarted","Data":"41eb3d29cc757dd43678d809000541589c0c32cf9956972b07195964b9a110db"} Nov 01 16:28:34.430590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.430529 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" podStartSLOduration=1.4305128 podStartE2EDuration="1.4305128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:34.430079724 +0000 UTC m=+2452.133245660" watchObservedRunningTime="2026-11-01 16:28:34.4305128 +0000 UTC m=+2452.133678742" Nov 01 16:28:34.452451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.452399 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" podStartSLOduration=1.452382366 podStartE2EDuration="1.452382366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:34.447428364 +0000 UTC m=+2452.150594304" watchObservedRunningTime="2026-11-01 16:28:34.452382366 +0000 UTC m=+2452.155548303" Nov 01 16:28:34.531951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.531866 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:34.947584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:34.947552 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:35.179775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:35.177417 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:35.406259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:35.405141 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:35.411787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:35.408111 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:28:35.411787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:35.408176 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 01 16:28:35.535904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:35.535857 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:35.950553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:35.950503 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 01 16:28:35.950744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:35.950582 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 01 16:28:36.143940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:36.143904 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:36.414776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:36.414068 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:28:36.414776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:36.414135 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 01 16:28:36.872178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:36.872142 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:36.947560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:36.947527 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 01 16:28:36.947830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:36.947806 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 01 16:28:37.065897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:37.065839 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" podUID="6297ca2d-b5a2-4e44-9e7b-cc5167735352" containerName="payload-processing" containerID="cri-o://47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f" gracePeriod=25 Nov 01 16:28:37.114791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:37.103846 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" podUID="e49adbdf-60f3-4185-b408-da51610072d1" containerName="payload-pre-processing" containerID="cri-o://7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768" gracePeriod=25 Nov 01 16:28:37.405872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:37.401062 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:28:37.410532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:37.410494 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 01 16:28:37.818938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:37.818826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:37.950236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:37.950202 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 01 16:28:37.950528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:37.950504 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 01 16:28:38.401555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:38.401078 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:28:38.401555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:38.401168 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 01 16:28:38.614220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:38.614183 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:38.948053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:38.948014 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 01 16:28:38.948456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:38.948081 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 01 16:28:39.400774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:39.400128 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" start-of-body= Nov 01 16:28:39.400774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:39.400193 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.11:15021/healthz/ready\": dial tcp 10.133.1.11:15021: connect: connection refused" Nov 01 16:28:39.519019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:39.518968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:39.948552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:39.948519 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 01 16:28:39.949014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:39.948572 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 01 16:28:40.113849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.113810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:40.115880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.115850 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr"] Nov 01 16:28:40.118029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.117981 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" podUID="80b38e9a-4645-4a5b-877d-a3228d10f008" containerName="maas-api" containerID="cri-o://b81d6596e7b563c148810aa20d341a3efb7d51da8c55fe2fccbe9b1e9a3e0b56" gracePeriod=30 Nov 01 16:28:40.207160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.207061 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" podUID="e49adbdf-60f3-4185-b408-da51610072d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:28:40.207497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.207465 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" podUID="6297ca2d-b5a2-4e44-9e7b-cc5167735352" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:28:40.481735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.481619 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:40.655860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.654933 4573 generic.go:356] "Generic (PLEG): container finished" podID="80b38e9a-4645-4a5b-877d-a3228d10f008" containerID="b81d6596e7b563c148810aa20d341a3efb7d51da8c55fe2fccbe9b1e9a3e0b56" exitCode=0 Nov 01 16:28:40.655860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.655107 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" event={"ID":"80b38e9a-4645-4a5b-877d-a3228d10f008","Type":"ContainerDied","Data":"b81d6596e7b563c148810aa20d341a3efb7d51da8c55fe2fccbe9b1e9a3e0b56"} Nov 01 16:28:40.703298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.703260 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:40.750606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.750535 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:28:40.808525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.808488 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:40.843969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.843938 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-maas-api-tls\") pod \"80b38e9a-4645-4a5b-877d-a3228d10f008\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " Nov 01 16:28:40.844177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.843983 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b38e9a-4645-4a5b-877d-a3228d10f008-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b38e9a-4645-4a5b-877d-a3228d10f008-openshift-service-ca\") pod \"80b38e9a-4645-4a5b-877d-a3228d10f008\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " Nov 01 16:28:40.844177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.844165 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b38e9a-4645-4a5b-877d-a3228d10f008-kube-api-access-hdgck\" (UniqueName: \"kubernetes.io/projected/80b38e9a-4645-4a5b-877d-a3228d10f008-kube-api-access-hdgck\") pod \"80b38e9a-4645-4a5b-877d-a3228d10f008\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " Nov 01 16:28:40.844337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.844318 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-metrics-certs\") pod \"80b38e9a-4645-4a5b-877d-a3228d10f008\" (UID: \"80b38e9a-4645-4a5b-877d-a3228d10f008\") " Nov 01 16:28:40.844469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.844447 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b38e9a-4645-4a5b-877d-a3228d10f008-openshift-service-ca" pod "80b38e9a-4645-4a5b-877d-a3228d10f008" (UID: "80b38e9a-4645-4a5b-877d-a3228d10f008"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:40.845052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.845030 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80b38e9a-4645-4a5b-877d-a3228d10f008-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:40.847055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.847029 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b38e9a-4645-4a5b-877d-a3228d10f008-kube-api-access-hdgck" pod "80b38e9a-4645-4a5b-877d-a3228d10f008" (UID: "80b38e9a-4645-4a5b-877d-a3228d10f008"). InnerVolumeSpecName "kube-api-access-hdgck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:40.847166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.847074 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-metrics-certs" pod "80b38e9a-4645-4a5b-877d-a3228d10f008" (UID: "80b38e9a-4645-4a5b-877d-a3228d10f008"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:40.847166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.847132 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-maas-api-tls" pod "80b38e9a-4645-4a5b-877d-a3228d10f008" (UID: "80b38e9a-4645-4a5b-877d-a3228d10f008"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:40.946981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.946944 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgck\" (UniqueName: \"kubernetes.io/projected/80b38e9a-4645-4a5b-877d-a3228d10f008-kube-api-access-hdgck\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:40.946981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.946983 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:40.947279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.946997 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80b38e9a-4645-4a5b-877d-a3228d10f008-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:40.947813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.947776 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Nov 01 16:28:40.947931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:40.947840 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Nov 01 16:28:41.193416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.193360 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:41.349092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.349025 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:41.350149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.349811 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:41.350710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.350687 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:41.665462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.665428 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" Nov 01 16:28:41.665639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.665460 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr" event={"ID":"80b38e9a-4645-4a5b-877d-a3228d10f008","Type":"ContainerDied","Data":"b0414faf40fd8e7de5a0a7899d3a71aca31cf0be6a6455b8c1c227769ced2d2d"} Nov 01 16:28:41.665639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.665515 4573 scope.go:122] "RemoveContainer" containerID="b81d6596e7b563c148810aa20d341a3efb7d51da8c55fe2fccbe9b1e9a3e0b56" Nov 01 16:28:41.692086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.692059 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr"] Nov 01 16:28:41.698194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.698169 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c78897-5658d89c6f-mnlrr"] Nov 01 16:28:41.953906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.953830 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:41.954287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.954268 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:41.955066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:41.955049 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:28:42.074518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:42.074486 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:42.468352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:42.468319 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b38e9a-4645-4a5b-877d-a3228d10f008" path="/var/lib/kubelet/pods/80b38e9a-4645-4a5b-877d-a3228d10f008/volumes" Nov 01 16:28:43.036576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:43.036536 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:43.988631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:43.988590 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:44.788410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:44.788375 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:45.561234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:45.561198 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:46.314040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:46.314004 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:47.174706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:47.174666 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:47.902201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:47.902167 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:47.998578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:47.998535 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:47.998962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:47.998936 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69"] Nov 01 16:28:48.158999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.158901 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:48.239509 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.239474 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7"] Nov 01 16:28:48.240355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.240315 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" containerID="cri-o://213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d" gracePeriod=30 Nov 01 16:28:48.277775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.275232 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x"] Nov 01 16:28:48.277775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.275464 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" containerID="cri-o://46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264" gracePeriod=30 Nov 01 16:28:48.278842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.278678 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:48.280643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.280612 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s"] Nov 01 16:28:48.281578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.281551 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" containerID="cri-o://fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db" gracePeriod=30 Nov 01 16:28:48.719560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.719524 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:48.844262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:48.844233 4573 scope.go:122] "RemoveContainer" containerID="7be0fbcd8582656a8d92d7fff31ad32d95a480b0dd7c4091e163cae414514b39" Nov 01 16:28:49.151478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:49.151445 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:49.912224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:49.912190 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9"] Nov 01 16:28:49.912733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:49.912440 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" podUID="72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" containerName="maas-api" containerID="cri-o://4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b" gracePeriod=30 Nov 01 16:28:49.949136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:49.949101 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:49.991903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:49.991869 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz"] Nov 01 16:28:49.992173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:49.992144 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" podUID="ba2255ef-523d-4c04-831c-5c07035a96ee" containerName="payload-processing" containerID="cri-o://8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f" gracePeriod=30 Nov 01 16:28:50.052665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.052587 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:50.149250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.149213 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:50.348736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.348291 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:50.432360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.398511 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" podUID="6297ca2d-b5a2-4e44-9e7b-cc5167735352" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:28:50.432360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.399053 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" podUID="e49adbdf-60f3-4185-b408-da51610072d1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:28:50.432360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.432154 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:50.520583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.520551 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:28:50.573792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.573746 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:50.576105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.576084 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:28:50.623328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.623301 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba2255ef-523d-4c04-831c-5c07035a96ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba2255ef-523d-4c04-831c-5c07035a96ee-plugins-config-volume\") pod \"ba2255ef-523d-4c04-831c-5c07035a96ee\" (UID: \"ba2255ef-523d-4c04-831c-5c07035a96ee\") " Nov 01 16:28:50.623467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.623367 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-maas-api-tls\") pod \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " Nov 01 16:28:50.623467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.623415 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-kube-api-access-pwps6\" (UniqueName: \"kubernetes.io/projected/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-kube-api-access-pwps6\") pod \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " Nov 01 16:28:50.623570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.623465 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba2255ef-523d-4c04-831c-5c07035a96ee-kube-api-access-mc629\" (UniqueName: \"kubernetes.io/projected/ba2255ef-523d-4c04-831c-5c07035a96ee-kube-api-access-mc629\") pod \"ba2255ef-523d-4c04-831c-5c07035a96ee\" (UID: \"ba2255ef-523d-4c04-831c-5c07035a96ee\") " Nov 01 16:28:50.623671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.623646 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-openshift-service-ca\") pod \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " Nov 01 16:28:50.623831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.623700 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba2255ef-523d-4c04-831c-5c07035a96ee-plugins-config-volume" pod "ba2255ef-523d-4c04-831c-5c07035a96ee" (UID: "ba2255ef-523d-4c04-831c-5c07035a96ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:50.623924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.623845 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-metrics-certs\") pod \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\" (UID: \"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe\") " Nov 01 16:28:50.624063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.624035 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-openshift-service-ca" pod "72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" (UID: "72a3a3e6-9ce5-48f2-a744-95b7aef0efbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:50.624602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.624577 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba2255ef-523d-4c04-831c-5c07035a96ee-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:50.624722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.624608 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:50.625715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.625689 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-maas-api-tls" pod "72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" (UID: "72a3a3e6-9ce5-48f2-a744-95b7aef0efbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:50.625904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.625885 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2255ef-523d-4c04-831c-5c07035a96ee-kube-api-access-mc629" pod "ba2255ef-523d-4c04-831c-5c07035a96ee" (UID: "ba2255ef-523d-4c04-831c-5c07035a96ee"). InnerVolumeSpecName "kube-api-access-mc629". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:50.626144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.626125 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-kube-api-access-pwps6" pod "72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" (UID: "72a3a3e6-9ce5-48f2-a744-95b7aef0efbe"). InnerVolumeSpecName "kube-api-access-pwps6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:50.626202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.626144 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-metrics-certs" pod "72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" (UID: "72a3a3e6-9ce5-48f2-a744-95b7aef0efbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:28:50.726617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.726534 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:50.726617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.726565 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:50.726617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.726575 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwps6\" (UniqueName: \"kubernetes.io/projected/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe-kube-api-access-pwps6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:50.726617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.726585 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc629\" (UniqueName: \"kubernetes.io/projected/ba2255ef-523d-4c04-831c-5c07035a96ee-kube-api-access-mc629\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:50.749621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.749589 4573 generic.go:356] "Generic (PLEG): container finished" podID="72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" containerID="4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b" exitCode=0 Nov 01 16:28:50.749830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.749639 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" Nov 01 16:28:50.749830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.749678 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" event={"ID":"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe","Type":"ContainerDied","Data":"4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b"} Nov 01 16:28:50.749830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.749811 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9" event={"ID":"72a3a3e6-9ce5-48f2-a744-95b7aef0efbe","Type":"ContainerDied","Data":"c308028acb101f8722cb82504cdb5d5a4bfaafea450d65d0d88127c941bcb95b"} Nov 01 16:28:50.750036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.749849 4573 scope.go:122] "RemoveContainer" containerID="4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b" Nov 01 16:28:50.751553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.751520 4573 generic.go:356] "Generic (PLEG): container finished" podID="ba2255ef-523d-4c04-831c-5c07035a96ee" containerID="8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f" exitCode=0 Nov 01 16:28:50.751659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.751578 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" Nov 01 16:28:50.751741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.751727 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" event={"ID":"ba2255ef-523d-4c04-831c-5c07035a96ee","Type":"ContainerDied","Data":"8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f"} Nov 01 16:28:50.751825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.751781 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz" event={"ID":"ba2255ef-523d-4c04-831c-5c07035a96ee","Type":"ContainerDied","Data":"4f120643a8b13e2a26460f05efd330fb8e2a0486da84636887b01ac769f0d412"} Nov 01 16:28:50.771206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.771184 4573 scope.go:122] "RemoveContainer" containerID="4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b" Nov 01 16:28:50.772200 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:50.772176 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b\": container with ID starting with 4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b not found: ID does not exist" containerID="4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b" Nov 01 16:28:50.772276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.772208 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b"} err="failed to get container status \"4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b\": rpc error: code = NotFound desc = could not find container \"4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b\": container with ID starting with 4987957acc7f211ba8568e8249c12ffba43e45b94f0a14285dac71105663a15b not found: ID does not exist" Nov 01 16:28:50.772276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.772228 4573 scope.go:122] "RemoveContainer" containerID="8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f" Nov 01 16:28:50.785073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.785053 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9"] Nov 01 16:28:50.785617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.785600 4573 scope.go:122] "RemoveContainer" containerID="8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f" Nov 01 16:28:50.785968 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:50.785942 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f\": container with ID starting with 8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f not found: ID does not exist" containerID="8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f" Nov 01 16:28:50.786054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.785977 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f"} err="failed to get container status \"8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f\": rpc error: code = NotFound desc = could not find container \"8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f\": container with ID starting with 8298b888f985c75652596ae123e7b2d2cb15fc112a1740aaf9739b5a6cb0963f not found: ID does not exist" Nov 01 16:28:50.790031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.790011 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-104b6e-65bf5bd66-jfjb9"] Nov 01 16:28:50.802631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.802610 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz"] Nov 01 16:28:50.809549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.809522 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-104b6e-776769677d-8h8mz"] Nov 01 16:28:50.954910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:50.954882 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:51.353698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:51.353668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:52.103418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:52.103387 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:52.460204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:52.460170 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" path="/var/lib/kubelet/pods/72a3a3e6-9ce5-48f2-a744-95b7aef0efbe/volumes" Nov 01 16:28:52.460642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:52.460625 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba2255ef-523d-4c04-831c-5c07035a96ee" path="/var/lib/kubelet/pods/ba2255ef-523d-4c04-831c-5c07035a96ee/volumes" Nov 01 16:28:52.679339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:52.679306 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn"] Nov 01 16:28:52.698362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:52.698332 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:53.169513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.169477 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:53.271028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.270998 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:53.599382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.599347 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:28:53.636117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.636094 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:53.651274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.651187 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:28:53.684802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684565 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-data\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.684802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684651 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-certs\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.684802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684689 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-token\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-token\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.684802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684779 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-kube-api-access-fp69p\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-kube-api-access-fp69p\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684850 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-certs\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684885 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d13c4b2-be53-42da-8450-88ccb0523bf1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d13c4b2-be53-42da-8450-88ccb0523bf1-istiod-ca-cert\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684939 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-socket\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684971 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-data\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.684995 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-certs\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.685024 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31dadf08-84bd-445e-85d7-f2795f22b4c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dadf08-84bd-445e-85d7-f2795f22b4c1-istiod-ca-cert\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.685066 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-kube-api-access-8n5m4\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-kube-api-access-8n5m4\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.685119 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-socket\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.685146 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-token\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-token\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.685183 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-envoy\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.685213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.685210 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe880f4e-50c1-40db-9900-87ac858f8e9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe880f4e-50c1-40db-9900-87ac858f8e9d-istiod-ca-cert\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.686155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686087 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-credential-socket\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.686292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686162 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-credential-socket\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.686292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686214 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-podinfo\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.686292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686254 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-podinfo\") pod \"2d13c4b2-be53-42da-8450-88ccb0523bf1\" (UID: \"2d13c4b2-be53-42da-8450-88ccb0523bf1\") " Nov 01 16:28:53.686292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686290 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-envoy\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.686473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686314 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-podinfo\") pod \"fe880f4e-50c1-40db-9900-87ac858f8e9d\" (UID: \"fe880f4e-50c1-40db-9900-87ac858f8e9d\") " Nov 01 16:28:53.686473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686337 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-token\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-token\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.686473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686371 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-kube-api-access-4s5gm\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-kube-api-access-4s5gm\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.686473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686391 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-credential-socket\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.686473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686420 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-data\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.686473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686450 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-socket\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.686813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.686522 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-envoy\") pod \"31dadf08-84bd-445e-85d7-f2795f22b4c1\" (UID: \"31dadf08-84bd-445e-85d7-f2795f22b4c1\") " Nov 01 16:28:53.688736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.687840 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-credential-socket" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.688736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.688093 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d13c4b2-be53-42da-8450-88ccb0523bf1-istiod-ca-cert" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:53.688736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.688323 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-data" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.688736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.688508 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-certs" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.689157 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe880f4e-50c1-40db-9900-87ac858f8e9d-istiod-ca-cert" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.689438 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-certs" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.689623 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-socket" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.689818 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-data" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.690073 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-credential-socket" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.690260 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-data" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.690454 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-credential-socket" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.693835 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-certs" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.694034 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-socket" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.694542 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-socket" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.695164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.694726 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dadf08-84bd-445e-85d7-f2795f22b4c1-istiod-ca-cert" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:53.697092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.696917 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-podinfo" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:53.697092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.696985 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-kube-api-access-fp69p" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "kube-api-access-fp69p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.697092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.697015 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-podinfo" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:53.697092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.697054 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-envoy" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.697413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.697149 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-kube-api-access-8n5m4" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "kube-api-access-8n5m4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.698065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.697929 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-token" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.698065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.698026 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-token" pod "2d13c4b2-be53-42da-8450-88ccb0523bf1" (UID: "2d13c4b2-be53-42da-8450-88ccb0523bf1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.699603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.699582 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-kube-api-access-4s5gm" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "kube-api-access-4s5gm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.703395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.702292 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-envoy" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.703395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.702399 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-token" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:53.715869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.715732 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-podinfo" pod "fe880f4e-50c1-40db-9900-87ac858f8e9d" (UID: "fe880f4e-50c1-40db-9900-87ac858f8e9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:28:53.717216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.717182 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-envoy" pod "31dadf08-84bd-445e-85d7-f2795f22b4c1" (UID: "31dadf08-84bd-445e-85d7-f2795f22b4c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:28:53.746354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.746308 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q"] Nov 01 16:28:53.781605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.781567 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:53.788868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788835 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.788868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788868 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe880f4e-50c1-40db-9900-87ac858f8e9d-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788886 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788902 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788917 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788931 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788944 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788958 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788972 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.788986 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5gm\" (UniqueName: \"kubernetes.io/projected/31dadf08-84bd-445e-85d7-f2795f22b4c1-kube-api-access-4s5gm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789001 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789015 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789029 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789043 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789056 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789069 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31dadf08-84bd-445e-85d7-f2795f22b4c1-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789082 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789095 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp69p\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-kube-api-access-fp69p\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789109 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789122 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d13c4b2-be53-42da-8450-88ccb0523bf1-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789135 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789149 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.789157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789162 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe880f4e-50c1-40db-9900-87ac858f8e9d-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789176 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31dadf08-84bd-445e-85d7-f2795f22b4c1-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789190 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n5m4\" (UniqueName: \"kubernetes.io/projected/2d13c4b2-be53-42da-8450-88ccb0523bf1-kube-api-access-8n5m4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789204 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d13c4b2-be53-42da-8450-88ccb0523bf1-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789218 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe880f4e-50c1-40db-9900-87ac858f8e9d-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789307 4573 generic.go:356] "Generic (PLEG): container finished" podID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerID="fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db" exitCode=0 Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789364 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789395 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" event={"ID":"31dadf08-84bd-445e-85d7-f2795f22b4c1","Type":"ContainerDied","Data":"fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db"} Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789451 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s" event={"ID":"31dadf08-84bd-445e-85d7-f2795f22b4c1","Type":"ContainerDied","Data":"21b9d456b42132c33973094852976eae0a94c830e4c73f4890947343a8f666e1"} Nov 01 16:28:53.790134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.789479 4573 scope.go:122] "RemoveContainer" containerID="fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db" Nov 01 16:28:53.795875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.795549 4573 generic.go:356] "Generic (PLEG): container finished" podID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerID="213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d" exitCode=0 Nov 01 16:28:53.795875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.795733 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" event={"ID":"2d13c4b2-be53-42da-8450-88ccb0523bf1","Type":"ContainerDied","Data":"213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d"} Nov 01 16:28:53.795875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.795815 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" Nov 01 16:28:53.795875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.795830 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7" event={"ID":"2d13c4b2-be53-42da-8450-88ccb0523bf1","Type":"ContainerDied","Data":"d5604ec4d0e4d4d40783f93f9c911559151c598d516b4770958b6c1e22ae11ec"} Nov 01 16:28:53.805471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.805453 4573 scope.go:122] "RemoveContainer" containerID="fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db" Nov 01 16:28:53.806035 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:53.805847 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db\": container with ID starting with fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db not found: ID does not exist" containerID="fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db" Nov 01 16:28:53.806217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.806191 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db"} err="failed to get container status \"fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db\": rpc error: code = NotFound desc = could not find container \"fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db\": container with ID starting with fc98ab56b1762964628ea75983358800a46f48218fc413b381fb74d6cfa132db not found: ID does not exist" Nov 01 16:28:53.806350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.806309 4573 scope.go:122] "RemoveContainer" containerID="213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d" Nov 01 16:28:53.807153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.807134 4573 generic.go:356] "Generic (PLEG): container finished" podID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerID="46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264" exitCode=0 Nov 01 16:28:53.807265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.807189 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" event={"ID":"fe880f4e-50c1-40db-9900-87ac858f8e9d","Type":"ContainerDied","Data":"46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264"} Nov 01 16:28:53.807265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.807234 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" event={"ID":"fe880f4e-50c1-40db-9900-87ac858f8e9d","Type":"ContainerDied","Data":"2e6fd629f7a33ffa8d778a35baabdaaf05432859ca05b145855ee61afe8a2b54"} Nov 01 16:28:53.807475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.807304 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x" Nov 01 16:28:53.827083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.827062 4573 scope.go:122] "RemoveContainer" containerID="213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d" Nov 01 16:28:53.828494 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:53.827368 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d\": container with ID starting with 213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d not found: ID does not exist" containerID="213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d" Nov 01 16:28:53.828494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.827397 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d"} err="failed to get container status \"213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d\": rpc error: code = NotFound desc = could not find container \"213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d\": container with ID starting with 213d566fb0beaf7c34fbffe6144ee3bdb7ee39daf2feb46f8f46245e62152e5d not found: ID does not exist" Nov 01 16:28:53.828494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.827417 4573 scope.go:122] "RemoveContainer" containerID="46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264" Nov 01 16:28:53.840174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.840151 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7"] Nov 01 16:28:53.842516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.842494 4573 scope.go:122] "RemoveContainer" containerID="46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264" Nov 01 16:28:53.842860 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:53.842840 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264\": container with ID starting with 46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264 not found: ID does not exist" containerID="46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264" Nov 01 16:28:53.842933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.842867 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264"} err="failed to get container status \"46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264\": rpc error: code = NotFound desc = could not find container \"46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264\": container with ID starting with 46e218e81efb9a7b0537ca623f7fc2aed6fae215413fb403e88096a1544b3264 not found: ID does not exist" Nov 01 16:28:53.848053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.848024 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-bvlp7"] Nov 01 16:28:53.926009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:53.887040 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x"] Nov 01 16:28:54.035864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.035769 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-v576x"] Nov 01 16:28:54.042585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.042542 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:54.062430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.051081 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s"] Nov 01 16:28:54.062430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.054686 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c78897-openshift-default-7c9787498f-79q4s"] Nov 01 16:28:54.139803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.138428 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:54.365175 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:54.365141 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:28:54.372115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.372091 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:54.480846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.480810 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" path="/var/lib/kubelet/pods/2d13c4b2-be53-42da-8450-88ccb0523bf1/volumes" Nov 01 16:28:54.481568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.481540 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" path="/var/lib/kubelet/pods/31dadf08-84bd-445e-85d7-f2795f22b4c1/volumes" Nov 01 16:28:54.482250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:54.482227 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" path="/var/lib/kubelet/pods/fe880f4e-50c1-40db-9900-87ac858f8e9d/volumes" Nov 01 16:28:55.170985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:55.170947 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:56.010597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:56.010555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:57.183792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.183196 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:57.312594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.312567 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:28:57.425834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.425800 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:57.426110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.425982 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6297ca2d-b5a2-4e44-9e7b-cc5167735352-kube-api-access-xswqz\" (UniqueName: \"kubernetes.io/projected/6297ca2d-b5a2-4e44-9e7b-cc5167735352-kube-api-access-xswqz\") pod \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\" (UID: \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\") " Nov 01 16:28:57.426344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.426205 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6297ca2d-b5a2-4e44-9e7b-cc5167735352-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6297ca2d-b5a2-4e44-9e7b-cc5167735352-plugins-config-volume\") pod \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\" (UID: \"6297ca2d-b5a2-4e44-9e7b-cc5167735352\") " Nov 01 16:28:57.426853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.426823 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6297ca2d-b5a2-4e44-9e7b-cc5167735352-plugins-config-volume" pod "6297ca2d-b5a2-4e44-9e7b-cc5167735352" (UID: "6297ca2d-b5a2-4e44-9e7b-cc5167735352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:57.427214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.427190 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6297ca2d-b5a2-4e44-9e7b-cc5167735352-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:57.428631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.428607 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6297ca2d-b5a2-4e44-9e7b-cc5167735352-kube-api-access-xswqz" pod "6297ca2d-b5a2-4e44-9e7b-cc5167735352" (UID: "6297ca2d-b5a2-4e44-9e7b-cc5167735352"). InnerVolumeSpecName "kube-api-access-xswqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:57.459678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.459654 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:28:57.534370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.534334 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49adbdf-60f3-4185-b408-da51610072d1-kube-api-access-qqmv8\" (UniqueName: \"kubernetes.io/projected/e49adbdf-60f3-4185-b408-da51610072d1-kube-api-access-qqmv8\") pod \"e49adbdf-60f3-4185-b408-da51610072d1\" (UID: \"e49adbdf-60f3-4185-b408-da51610072d1\") " Nov 01 16:28:57.534886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.534855 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49adbdf-60f3-4185-b408-da51610072d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49adbdf-60f3-4185-b408-da51610072d1-plugins-config-volume\") pod \"e49adbdf-60f3-4185-b408-da51610072d1\" (UID: \"e49adbdf-60f3-4185-b408-da51610072d1\") " Nov 01 16:28:57.538904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.535661 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xswqz\" (UniqueName: \"kubernetes.io/projected/6297ca2d-b5a2-4e44-9e7b-cc5167735352-kube-api-access-xswqz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:57.538904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.536113 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49adbdf-60f3-4185-b408-da51610072d1-plugins-config-volume" pod "e49adbdf-60f3-4185-b408-da51610072d1" (UID: "e49adbdf-60f3-4185-b408-da51610072d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:28:57.538904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.536879 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:57.541281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.541255 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49adbdf-60f3-4185-b408-da51610072d1-kube-api-access-qqmv8" pod "e49adbdf-60f3-4185-b408-da51610072d1" (UID: "e49adbdf-60f3-4185-b408-da51610072d1"). InnerVolumeSpecName "kube-api-access-qqmv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:28:57.636730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.636688 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:57.636935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.636910 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e49adbdf-60f3-4185-b408-da51610072d1-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:57.636935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.636932 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmv8\" (UniqueName: \"kubernetes.io/projected/e49adbdf-60f3-4185-b408-da51610072d1-kube-api-access-qqmv8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:28:57.742155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.742056 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:57.842742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.842704 4573 generic.go:356] "Generic (PLEG): container finished" podID="e49adbdf-60f3-4185-b408-da51610072d1" containerID="7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768" exitCode=0 Nov 01 16:28:57.842952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.842781 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" Nov 01 16:28:57.843817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.842785 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" event={"ID":"e49adbdf-60f3-4185-b408-da51610072d1","Type":"ContainerDied","Data":"7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768"} Nov 01 16:28:57.843972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.843855 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb" event={"ID":"e49adbdf-60f3-4185-b408-da51610072d1","Type":"ContainerDied","Data":"d86aea33a95d429e72808654e935613cd2af51c6ce292f74fa377fda642d2310"} Nov 01 16:28:57.843972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.843889 4573 scope.go:122] "RemoveContainer" containerID="7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768" Nov 01 16:28:57.844803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.844781 4573 generic.go:356] "Generic (PLEG): container finished" podID="6297ca2d-b5a2-4e44-9e7b-cc5167735352" containerID="47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f" exitCode=0 Nov 01 16:28:57.844987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.844970 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" Nov 01 16:28:57.845153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.845065 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" event={"ID":"6297ca2d-b5a2-4e44-9e7b-cc5167735352","Type":"ContainerDied","Data":"47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f"} Nov 01 16:28:57.845153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.845111 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt" event={"ID":"6297ca2d-b5a2-4e44-9e7b-cc5167735352","Type":"ContainerDied","Data":"a9352585db6b376676e42d65dad887ab793ac1119e6d391b6cb5c17b7256f7e8"} Nov 01 16:28:57.927431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.926169 4573 scope.go:122] "RemoveContainer" containerID="7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768" Nov 01 16:28:57.927711 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:57.927685 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768\": container with ID starting with 7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768 not found: ID does not exist" containerID="7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768" Nov 01 16:28:57.928179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.927854 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768"} err="failed to get container status \"7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768\": rpc error: code = NotFound desc = could not find container \"7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768\": container with ID starting with 7142ad89daaf64ff262888ea13a98acba5738f0b5db846087342a13c8d7b3768 not found: ID does not exist" Nov 01 16:28:57.928179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.927884 4573 scope.go:122] "RemoveContainer" containerID="47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f" Nov 01 16:28:57.969586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:57.969554 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb"] Nov 01 16:28:58.013034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.012993 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:58.021583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.021551 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-c78897-86d6f6596c-74kkb"] Nov 01 16:28:58.028321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.028250 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt"] Nov 01 16:28:58.035827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.033986 4573 scope.go:122] "RemoveContainer" containerID="47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f" Nov 01 16:28:58.035827 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:28:58.034354 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f\": container with ID starting with 47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f not found: ID does not exist" containerID="47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f" Nov 01 16:28:58.035827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.034387 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f"} err="failed to get container status \"47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f\": rpc error: code = NotFound desc = could not find container \"47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f\": container with ID starting with 47c80a01aa07a59928c278d654e859eddda2d20e466cadb56000e90a63f6d30f not found: ID does not exist" Nov 01 16:28:58.035827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.035560 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-c78897-68dd595655-bgrvt"] Nov 01 16:28:58.299742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.299663 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs"] Nov 01 16:28:58.300378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300363 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e49adbdf-60f3-4185-b408-da51610072d1" containerName="payload-pre-processing" Nov 01 16:28:58.300439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300378 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49adbdf-60f3-4185-b408-da51610072d1" containerName="payload-pre-processing" Nov 01 16:28:58.300439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300403 4573 cpu_manager.go:414] "containerMap: removing container" podUID="72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" containerName="maas-api" Nov 01 16:28:58.300439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300409 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" containerName="maas-api" Nov 01 16:28:58.300439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300424 4573 cpu_manager.go:414] "containerMap: removing container" podUID="80b38e9a-4645-4a5b-877d-a3228d10f008" containerName="maas-api" Nov 01 16:28:58.300439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300429 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b38e9a-4645-4a5b-877d-a3228d10f008" containerName="maas-api" Nov 01 16:28:58.300439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300436 4573 cpu_manager.go:414] "containerMap: removing container" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" Nov 01 16:28:58.300439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300441 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300447 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300452 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300468 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ba2255ef-523d-4c04-831c-5c07035a96ee" containerName="payload-processing" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300472 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba2255ef-523d-4c04-831c-5c07035a96ee" containerName="payload-processing" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300481 4573 cpu_manager.go:414] "containerMap: removing container" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300486 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300495 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6297ca2d-b5a2-4e44-9e7b-cc5167735352" containerName="payload-processing" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300500 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6297ca2d-b5a2-4e44-9e7b-cc5167735352" containerName="payload-processing" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300597 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49adbdf-60f3-4185-b408-da51610072d1" containerName="payload-pre-processing" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300610 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="72a3a3e6-9ce5-48f2-a744-95b7aef0efbe" containerName="maas-api" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300618 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b38e9a-4645-4a5b-877d-a3228d10f008" containerName="maas-api" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300622 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe880f4e-50c1-40db-9900-87ac858f8e9d" containerName="istio-proxy" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300628 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d13c4b2-be53-42da-8450-88ccb0523bf1" containerName="istio-proxy" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300635 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba2255ef-523d-4c04-831c-5c07035a96ee" containerName="payload-processing" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300642 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dadf08-84bd-445e-85d7-f2795f22b4c1" containerName="istio-proxy" Nov 01 16:28:58.300669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.300648 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6297ca2d-b5a2-4e44-9e7b-cc5167735352" containerName="payload-processing" Nov 01 16:28:58.305709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.305690 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.317807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.315344 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs"] Nov 01 16:28:58.349107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.349065 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:58.417251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.417217 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:28:58.417461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.417321 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:28:58.422714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.422558 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.423596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.423098 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.423596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.423149 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lkjg\" (UniqueName: \"kubernetes.io/projected/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-kube-api-access-7lkjg\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.461843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.461807 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6297ca2d-b5a2-4e44-9e7b-cc5167735352" path="/var/lib/kubelet/pods/6297ca2d-b5a2-4e44-9e7b-cc5167735352/volumes" Nov 01 16:28:58.462391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.462375 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49adbdf-60f3-4185-b408-da51610072d1" path="/var/lib/kubelet/pods/e49adbdf-60f3-4185-b408-da51610072d1/volumes" Nov 01 16:28:58.524290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.524245 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.524290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.524298 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lkjg\" (UniqueName: \"kubernetes.io/projected/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-kube-api-access-7lkjg\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.524517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.524475 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.524984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.524959 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.526896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.526875 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.536582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.536559 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lkjg\" (UniqueName: \"kubernetes.io/projected/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-kube-api-access-7lkjg\") pod \"maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.625870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.625832 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:28:58.767394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.767360 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:58.818561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.818526 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58"] Nov 01 16:28:58.824566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.824543 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:58.831132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.831107 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fd73f-dockercfg-mgdgh\"" Nov 01 16:28:58.831985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.831964 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fd73f\"" Nov 01 16:28:58.834306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.834285 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58"] Nov 01 16:28:58.838695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.838673 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl"] Nov 01 16:28:58.844003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.843979 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:58.854913 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.854889 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl"] Nov 01 16:28:58.869993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.869950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:58.910840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.910809 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs"] Nov 01 16:28:58.911293 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:28:58.911272 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d22278b_8fe9_401d_bbf6_fe96c5b443c8.slice/crio-d944817bfe5476dc8e3a42d5f3c62ddab5c17b457426b722366c7602224fd6f9 WatchSource:0}: Error finding container d944817bfe5476dc8e3a42d5f3c62ddab5c17b457426b722366c7602224fd6f9: Status 404 returned error can't find the container with id d944817bfe5476dc8e3a42d5f3c62ddab5c17b457426b722366c7602224fd6f9 Nov 01 16:28:58.932691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.932659 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc8mb\" (UniqueName: \"kubernetes.io/projected/bf69a1d6-83d6-40f9-b798-5502d0b380e0-kube-api-access-fc8mb\") pod \"payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58\" (UID: \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:58.932895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.932783 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf69a1d6-83d6-40f9-b798-5502d0b380e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58\" (UID: \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:58.933005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.932924 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86149c0b-b1a1-4b7b-a618-eb47b706771d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl\" (UID: \"86149c0b-b1a1-4b7b-a618-eb47b706771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:58.933005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:58.932971 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/86149c0b-b1a1-4b7b-a618-eb47b706771d-kube-api-access-8b4ws\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl\" (UID: \"86149c0b-b1a1-4b7b-a618-eb47b706771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:59.037581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.037527 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8mb\" (UniqueName: \"kubernetes.io/projected/bf69a1d6-83d6-40f9-b798-5502d0b380e0-kube-api-access-fc8mb\") pod \"payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58\" (UID: \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:59.037839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.037685 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf69a1d6-83d6-40f9-b798-5502d0b380e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58\" (UID: \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:59.053783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.052223 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86149c0b-b1a1-4b7b-a618-eb47b706771d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl\" (UID: \"86149c0b-b1a1-4b7b-a618-eb47b706771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:59.053783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.052293 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/86149c0b-b1a1-4b7b-a618-eb47b706771d-kube-api-access-8b4ws\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl\" (UID: \"86149c0b-b1a1-4b7b-a618-eb47b706771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:59.053783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.053601 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf69a1d6-83d6-40f9-b798-5502d0b380e0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58\" (UID: \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:59.057781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.054421 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86149c0b-b1a1-4b7b-a618-eb47b706771d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl\" (UID: \"86149c0b-b1a1-4b7b-a618-eb47b706771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:59.077163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.075573 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8mb\" (UniqueName: \"kubernetes.io/projected/bf69a1d6-83d6-40f9-b798-5502d0b380e0-kube-api-access-fc8mb\") pod \"payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58\" (UID: \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:59.084371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.084298 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/86149c0b-b1a1-4b7b-a618-eb47b706771d-kube-api-access-8b4ws\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl\" (UID: \"86149c0b-b1a1-4b7b-a618-eb47b706771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:59.143574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.143534 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:59.169724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.169697 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:59.304771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.297963 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh"] Nov 01 16:28:59.313135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.313102 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p"] Nov 01 16:28:59.359203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.359163 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:59.455212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.455186 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58"] Nov 01 16:28:59.455682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.455659 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:59.483252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.481521 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl"] Nov 01 16:28:59.485573 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:28:59.485542 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86149c0b_b1a1_4b7b_a618_eb47b706771d.slice/crio-60a6d2c357eb81912f167bc8bd058b16a624908850021d5d8cef8e031b99d89e WatchSource:0}: Error finding container 60a6d2c357eb81912f167bc8bd058b16a624908850021d5d8cef8e031b99d89e: Status 404 returned error can't find the container with id 60a6d2c357eb81912f167bc8bd058b16a624908850021d5d8cef8e031b99d89e Nov 01 16:28:59.692776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.691985 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:28:59.875152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.875113 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" event={"ID":"2d22278b-8fe9-401d-bbf6-fe96c5b443c8","Type":"ContainerStarted","Data":"543e2b19108e1d506e7c7029f91e381e65f86d15ed17e730cf67e2995bff750a"} Nov 01 16:28:59.875405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.875169 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" event={"ID":"2d22278b-8fe9-401d-bbf6-fe96c5b443c8","Type":"ContainerStarted","Data":"d944817bfe5476dc8e3a42d5f3c62ddab5c17b457426b722366c7602224fd6f9"} Nov 01 16:28:59.877055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.877022 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" event={"ID":"86149c0b-b1a1-4b7b-a618-eb47b706771d","Type":"ContainerStarted","Data":"242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3"} Nov 01 16:28:59.877203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.877076 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" event={"ID":"86149c0b-b1a1-4b7b-a618-eb47b706771d","Type":"ContainerStarted","Data":"60a6d2c357eb81912f167bc8bd058b16a624908850021d5d8cef8e031b99d89e"} Nov 01 16:28:59.877203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.877106 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:28:59.878634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.878610 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" event={"ID":"bf69a1d6-83d6-40f9-b798-5502d0b380e0","Type":"ContainerStarted","Data":"3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2"} Nov 01 16:28:59.878799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.878651 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" event={"ID":"bf69a1d6-83d6-40f9-b798-5502d0b380e0","Type":"ContainerStarted","Data":"efd3d379c03b426356fae844bcff92e7e9d6ae2389a53ea7e312189609fa15a9"} Nov 01 16:28:59.878799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.878681 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:28:59.894685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.894641 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" podStartSLOduration=1.894629281 podStartE2EDuration="1.894629281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:59.891884794 +0000 UTC m=+2477.595050729" watchObservedRunningTime="2026-11-01 16:28:59.894629281 +0000 UTC m=+2477.597795217" Nov 01 16:28:59.925052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.924992 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" podStartSLOduration=1.9249804830000001 podStartE2EDuration="1.924980483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:59.921744056 +0000 UTC m=+2477.624909992" watchObservedRunningTime="2026-11-01 16:28:59.924980483 +0000 UTC m=+2477.628146421" Nov 01 16:28:59.939458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:28:59.939411 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" podStartSLOduration=1.939398514 podStartE2EDuration="1.939398514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:28:59.938183638 +0000 UTC m=+2477.641349575" watchObservedRunningTime="2026-11-01 16:28:59.939398514 +0000 UTC m=+2477.642564486" Nov 01 16:29:00.585039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:00.584957 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:00.818892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:00.818833 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:00.889170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:00.889143 4573 generic.go:356] "Generic (PLEG): container finished" podID="2d22278b-8fe9-401d-bbf6-fe96c5b443c8" containerID="543e2b19108e1d506e7c7029f91e381e65f86d15ed17e730cf67e2995bff750a" exitCode=0 Nov 01 16:29:00.889329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:00.889222 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" event={"ID":"2d22278b-8fe9-401d-bbf6-fe96c5b443c8","Type":"ContainerDied","Data":"543e2b19108e1d506e7c7029f91e381e65f86d15ed17e730cf67e2995bff750a"} Nov 01 16:29:01.275199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:01.275124 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:02.061640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.061606 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:02.081581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.081555 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:29:02.158033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.157233 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-xks-ca\") pod \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " Nov 01 16:29:02.158033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.157419 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-kube-api-access-7lkjg\" (UniqueName: \"kubernetes.io/projected/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-kube-api-access-7lkjg\") pod \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " Nov 01 16:29:02.158033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.157626 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-openshift-ca\") pod \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\" (UID: \"2d22278b-8fe9-401d-bbf6-fe96c5b443c8\") " Nov 01 16:29:02.160224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.158808 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-openshift-ca" pod "2d22278b-8fe9-401d-bbf6-fe96c5b443c8" (UID: "2d22278b-8fe9-401d-bbf6-fe96c5b443c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:02.172899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.172845 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-xks-ca" pod "2d22278b-8fe9-401d-bbf6-fe96c5b443c8" (UID: "2d22278b-8fe9-401d-bbf6-fe96c5b443c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:02.173179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.172983 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-kube-api-access-7lkjg" pod "2d22278b-8fe9-401d-bbf6-fe96c5b443c8" (UID: "2d22278b-8fe9-401d-bbf6-fe96c5b443c8"). InnerVolumeSpecName "kube-api-access-7lkjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:02.259574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.259538 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lkjg\" (UniqueName: \"kubernetes.io/projected/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-kube-api-access-7lkjg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:02.259574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.259569 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:02.259574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.259581 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d22278b-8fe9-401d-bbf6-fe96c5b443c8-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:02.848348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.848265 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:02.923565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.923525 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" event={"ID":"2d22278b-8fe9-401d-bbf6-fe96c5b443c8","Type":"ContainerDied","Data":"d944817bfe5476dc8e3a42d5f3c62ddab5c17b457426b722366c7602224fd6f9"} Nov 01 16:29:02.923869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.923845 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d944817bfe5476dc8e3a42d5f3c62ddab5c17b457426b722366c7602224fd6f9" Nov 01 16:29:02.924065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:02.924052 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs" Nov 01 16:29:03.021686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.021643 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" podUID="24d3c991-2024-4b8f-b402-9d217856f58b" containerName="main" containerID="cri-o://6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660" gracePeriod=45 Nov 01 16:29:03.049592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.049550 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" podUID="24d3c991-2024-4b8f-b402-9d217856f58b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.232:8000/ready\": dial tcp 10.133.0.232:8000: connect: connection refused" Nov 01 16:29:03.318247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.318221 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:29:03.477116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477036 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-dshm\") pod \"24d3c991-2024-4b8f-b402-9d217856f58b\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " Nov 01 16:29:03.477116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477111 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-home\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-home\") pod \"24d3c991-2024-4b8f-b402-9d217856f58b\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " Nov 01 16:29:03.477319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477161 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d3c991-2024-4b8f-b402-9d217856f58b-tls-certs\" (UniqueName: \"kubernetes.io/secret/24d3c991-2024-4b8f-b402-9d217856f58b-tls-certs\") pod \"24d3c991-2024-4b8f-b402-9d217856f58b\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " Nov 01 16:29:03.477319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477225 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-model-cache\") pod \"24d3c991-2024-4b8f-b402-9d217856f58b\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " Nov 01 16:29:03.477319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477284 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-tmp-dir\") pod \"24d3c991-2024-4b8f-b402-9d217856f58b\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " Nov 01 16:29:03.477476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477398 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d3c991-2024-4b8f-b402-9d217856f58b-kube-api-access-xnkcg\" (UniqueName: \"kubernetes.io/projected/24d3c991-2024-4b8f-b402-9d217856f58b-kube-api-access-xnkcg\") pod \"24d3c991-2024-4b8f-b402-9d217856f58b\" (UID: \"24d3c991-2024-4b8f-b402-9d217856f58b\") " Nov 01 16:29:03.477476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477397 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-home" pod "24d3c991-2024-4b8f-b402-9d217856f58b" (UID: "24d3c991-2024-4b8f-b402-9d217856f58b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.477584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477525 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-model-cache" pod "24d3c991-2024-4b8f-b402-9d217856f58b" (UID: "24d3c991-2024-4b8f-b402-9d217856f58b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.477666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.477634 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-tmp-dir" pod "24d3c991-2024-4b8f-b402-9d217856f58b" (UID: "24d3c991-2024-4b8f-b402-9d217856f58b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.478143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.478118 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.478143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.478139 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.478333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.478149 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.479590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.479559 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d3c991-2024-4b8f-b402-9d217856f58b-tls-certs" pod "24d3c991-2024-4b8f-b402-9d217856f58b" (UID: "24d3c991-2024-4b8f-b402-9d217856f58b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:03.479590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.479566 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d3c991-2024-4b8f-b402-9d217856f58b-kube-api-access-xnkcg" pod "24d3c991-2024-4b8f-b402-9d217856f58b" (UID: "24d3c991-2024-4b8f-b402-9d217856f58b"). InnerVolumeSpecName "kube-api-access-xnkcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:03.479716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.479608 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-dshm" pod "24d3c991-2024-4b8f-b402-9d217856f58b" (UID: "24d3c991-2024-4b8f-b402-9d217856f58b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:03.585792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.580151 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnkcg\" (UniqueName: \"kubernetes.io/projected/24d3c991-2024-4b8f-b402-9d217856f58b-kube-api-access-xnkcg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.585792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.580310 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/24d3c991-2024-4b8f-b402-9d217856f58b-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.585792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.580331 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24d3c991-2024-4b8f-b402-9d217856f58b-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:03.867936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.867901 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56"] Nov 01 16:29:03.868639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.868624 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2d22278b-8fe9-401d-bbf6-fe96c5b443c8" containerName="revoke-keys" Nov 01 16:29:03.868685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.868641 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d22278b-8fe9-401d-bbf6-fe96c5b443c8" containerName="revoke-keys" Nov 01 16:29:03.868685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.868681 4573 cpu_manager.go:414] "containerMap: removing container" podUID="24d3c991-2024-4b8f-b402-9d217856f58b" containerName="main" Nov 01 16:29:03.868787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.868686 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d3c991-2024-4b8f-b402-9d217856f58b" containerName="main" Nov 01 16:29:03.868823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.868804 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d3c991-2024-4b8f-b402-9d217856f58b" containerName="main" Nov 01 16:29:03.868823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.868818 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d22278b-8fe9-401d-bbf6-fe96c5b443c8" containerName="revoke-keys" Nov 01 16:29:03.872813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.872798 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:03.907054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.907025 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:29:03.907229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.907219 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:29:03.913985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.913950 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222"] Nov 01 16:29:03.918602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.918577 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56"] Nov 01 16:29:03.918795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.918664 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:03.942888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.942860 4573 generic.go:356] "Generic (PLEG): container finished" podID="24d3c991-2024-4b8f-b402-9d217856f58b" containerID="6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660" exitCode=0 Nov 01 16:29:03.943045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.943028 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" event={"ID":"24d3c991-2024-4b8f-b402-9d217856f58b","Type":"ContainerDied","Data":"6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660"} Nov 01 16:29:03.943137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.943066 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" event={"ID":"24d3c991-2024-4b8f-b402-9d217856f58b","Type":"ContainerDied","Data":"2e9d6d754c1634e430ec8c21da204f7f58862530811094f5612615aae7242151"} Nov 01 16:29:03.943137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.943091 4573 scope.go:122] "RemoveContainer" containerID="6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660" Nov 01 16:29:03.943245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.943138 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69" Nov 01 16:29:03.951801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.951766 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222"] Nov 01 16:29:03.966796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.966768 4573 scope.go:122] "RemoveContainer" containerID="6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660" Nov 01 16:29:03.967217 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:03.967185 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660\": container with ID starting with 6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660 not found: ID does not exist" containerID="6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660" Nov 01 16:29:03.967311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.967229 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660"} err="failed to get container status \"6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660\": rpc error: code = NotFound desc = could not find container \"6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660\": container with ID starting with 6c67286cf385ed25b47c3f49e3dadd5c50d888485ce2649981c6c2d381ee0660 not found: ID does not exist" Nov 01 16:29:03.985580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.985554 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:03.985699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.985681 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4xqd\" (UniqueName: \"kubernetes.io/projected/8927f4c4-a345-48c1-bbb7-c231c39c4952-kube-api-access-d4xqd\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:03.985814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:03.985795 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:04.010185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.010154 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69"] Nov 01 16:29:04.021348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.021312 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-f47890/rate-test-model-f47890-kserve-784796cc45-6gr69"] Nov 01 16:29:04.086955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.086921 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab785806-a92a-49bb-a609-653301fc203a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.087129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.087048 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkzqs\" (UniqueName: \"kubernetes.io/projected/ab785806-a92a-49bb-a609-653301fc203a-kube-api-access-rkzqs\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.087129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.087105 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab785806-a92a-49bb-a609-653301fc203a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.087270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.087180 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:04.087350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.087335 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4xqd\" (UniqueName: \"kubernetes.io/projected/8927f4c4-a345-48c1-bbb7-c231c39c4952-kube-api-access-d4xqd\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:04.087410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.087394 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:04.088316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.088241 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:04.089848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.089826 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:04.095225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.095202 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4xqd\" (UniqueName: \"kubernetes.io/projected/8927f4c4-a345-48c1-bbb7-c231c39c4952-kube-api-access-d4xqd\") pod \"maas-api-revoke-sub-7c238f07-8ml56\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:04.131472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.131415 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:04.190765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.187709 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:04.190765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.188795 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab785806-a92a-49bb-a609-653301fc203a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.190765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.188886 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkzqs\" (UniqueName: \"kubernetes.io/projected/ab785806-a92a-49bb-a609-653301fc203a-kube-api-access-rkzqs\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.190765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.188930 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab785806-a92a-49bb-a609-653301fc203a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.190765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.190055 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab785806-a92a-49bb-a609-653301fc203a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.194074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.194040 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab785806-a92a-49bb-a609-653301fc203a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.204693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.204666 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkzqs\" (UniqueName: \"kubernetes.io/projected/ab785806-a92a-49bb-a609-653301fc203a-kube-api-access-rkzqs\") pod \"maas-api-revoke-sub-35da1e14-f5222\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.243836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.243805 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:04.299926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.299883 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" podUID="915836fc-5724-4713-a688-dc226a3e7818" containerName="payload-processing" containerID="cri-o://7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf" gracePeriod=25 Nov 01 16:29:04.316877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.316837 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" podUID="46336b63-f773-4697-a916-8adf998d6da2" containerName="payload-pre-processing" containerID="cri-o://f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9" gracePeriod=25 Nov 01 16:29:04.461655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.461583 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d3c991-2024-4b8f-b402-9d217856f58b" path="/var/lib/kubelet/pods/24d3c991-2024-4b8f-b402-9d217856f58b/volumes" Nov 01 16:29:04.476526 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:29:04.476496 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8927f4c4_a345_48c1_bbb7_c231c39c4952.slice/crio-ceeb7d424369130c117635f23fad732943b633a9de7166ae24bf8a312fba8cd9 WatchSource:0}: Error finding container ceeb7d424369130c117635f23fad732943b633a9de7166ae24bf8a312fba8cd9: Status 404 returned error can't find the container with id ceeb7d424369130c117635f23fad732943b633a9de7166ae24bf8a312fba8cd9 Nov 01 16:29:04.476906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.476884 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56"] Nov 01 16:29:04.541945 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:29:04.541912 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab785806_a92a_49bb_a609_653301fc203a.slice/crio-48e28ba5e2d0db6bb11b7d8f3f754b9f1e7148e11122bbf6e01a92fd9620f4cc WatchSource:0}: Error finding container 48e28ba5e2d0db6bb11b7d8f3f754b9f1e7148e11122bbf6e01a92fd9620f4cc: Status 404 returned error can't find the container with id 48e28ba5e2d0db6bb11b7d8f3f754b9f1e7148e11122bbf6e01a92fd9620f4cc Nov 01 16:29:04.542542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.542508 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222"] Nov 01 16:29:04.844054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.843956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:04.957075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.957030 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" event={"ID":"8927f4c4-a345-48c1-bbb7-c231c39c4952","Type":"ContainerStarted","Data":"e5bd8c4e306b992fec497e8ab5bfc52053dffdd5aba1ac8468d0ef0b2676b1e8"} Nov 01 16:29:04.957287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.957100 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" event={"ID":"8927f4c4-a345-48c1-bbb7-c231c39c4952","Type":"ContainerStarted","Data":"ceeb7d424369130c117635f23fad732943b633a9de7166ae24bf8a312fba8cd9"} Nov 01 16:29:04.958611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.958583 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" event={"ID":"ab785806-a92a-49bb-a609-653301fc203a","Type":"ContainerStarted","Data":"48e28ba5e2d0db6bb11b7d8f3f754b9f1e7148e11122bbf6e01a92fd9620f4cc"} Nov 01 16:29:04.974028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:04.973984 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" podStartSLOduration=1.973970319 podStartE2EDuration="1.973970319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:04.970410602 +0000 UTC m=+2482.673576539" watchObservedRunningTime="2026-11-01 16:29:04.973970319 +0000 UTC m=+2482.677136250" Nov 01 16:29:05.521732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:05.521695 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:05.969667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:05.969633 4573 generic.go:356] "Generic (PLEG): container finished" podID="ab785806-a92a-49bb-a609-653301fc203a" containerID="6549f513835043a61f4f4878d192d04c9ac8ab0823537e2d808556986bf60d6a" exitCode=0 Nov 01 16:29:05.969878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:05.969672 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" event={"ID":"ab785806-a92a-49bb-a609-653301fc203a","Type":"ContainerDied","Data":"6549f513835043a61f4f4878d192d04c9ac8ab0823537e2d808556986bf60d6a"} Nov 01 16:29:05.971889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:05.971865 4573 generic.go:356] "Generic (PLEG): container finished" podID="8927f4c4-a345-48c1-bbb7-c231c39c4952" containerID="e5bd8c4e306b992fec497e8ab5bfc52053dffdd5aba1ac8468d0ef0b2676b1e8" exitCode=0 Nov 01 16:29:05.972052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:05.971901 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" event={"ID":"8927f4c4-a345-48c1-bbb7-c231c39c4952","Type":"ContainerDied","Data":"e5bd8c4e306b992fec497e8ab5bfc52053dffdd5aba1ac8468d0ef0b2676b1e8"} Nov 01 16:29:06.351927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:06.351844 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:07.141312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.141248 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:07.262564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.262090 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:07.365846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.365812 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:07.456871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.423800 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab785806-a92a-49bb-a609-653301fc203a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab785806-a92a-49bb-a609-653301fc203a-maas-api-openshift-ca\") pod \"ab785806-a92a-49bb-a609-653301fc203a\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " Nov 01 16:29:07.456871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.424008 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab785806-a92a-49bb-a609-653301fc203a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab785806-a92a-49bb-a609-653301fc203a-maas-api-xks-ca\") pod \"ab785806-a92a-49bb-a609-653301fc203a\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " Nov 01 16:29:07.456871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.424150 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab785806-a92a-49bb-a609-653301fc203a-kube-api-access-rkzqs\" (UniqueName: \"kubernetes.io/projected/ab785806-a92a-49bb-a609-653301fc203a-kube-api-access-rkzqs\") pod \"ab785806-a92a-49bb-a609-653301fc203a\" (UID: \"ab785806-a92a-49bb-a609-653301fc203a\") " Nov 01 16:29:07.589100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.589006 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab785806-a92a-49bb-a609-653301fc203a-maas-api-openshift-ca" pod "ab785806-a92a-49bb-a609-653301fc203a" (UID: "ab785806-a92a-49bb-a609-653301fc203a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:07.612167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.611785 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-openshift-ca\") pod \"8927f4c4-a345-48c1-bbb7-c231c39c4952\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " Nov 01 16:29:07.612167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.611895 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-xks-ca\") pod \"8927f4c4-a345-48c1-bbb7-c231c39c4952\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " Nov 01 16:29:07.612167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.611950 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8927f4c4-a345-48c1-bbb7-c231c39c4952-kube-api-access-d4xqd\" (UniqueName: \"kubernetes.io/projected/8927f4c4-a345-48c1-bbb7-c231c39c4952-kube-api-access-d4xqd\") pod \"8927f4c4-a345-48c1-bbb7-c231c39c4952\" (UID: \"8927f4c4-a345-48c1-bbb7-c231c39c4952\") " Nov 01 16:29:07.613563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.613227 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab785806-a92a-49bb-a609-653301fc203a-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:07.613563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.613439 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-openshift-ca" pod "8927f4c4-a345-48c1-bbb7-c231c39c4952" (UID: "8927f4c4-a345-48c1-bbb7-c231c39c4952"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:07.633397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.631937 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab785806-a92a-49bb-a609-653301fc203a-kube-api-access-rkzqs" pod "ab785806-a92a-49bb-a609-653301fc203a" (UID: "ab785806-a92a-49bb-a609-653301fc203a"). InnerVolumeSpecName "kube-api-access-rkzqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:07.633982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.633928 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8927f4c4-a345-48c1-bbb7-c231c39c4952-kube-api-access-d4xqd" pod "8927f4c4-a345-48c1-bbb7-c231c39c4952" (UID: "8927f4c4-a345-48c1-bbb7-c231c39c4952"). InnerVolumeSpecName "kube-api-access-d4xqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:07.634973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.634939 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-xks-ca" pod "8927f4c4-a345-48c1-bbb7-c231c39c4952" (UID: "8927f4c4-a345-48c1-bbb7-c231c39c4952"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:07.649923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.649897 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab785806-a92a-49bb-a609-653301fc203a-maas-api-xks-ca" pod "ab785806-a92a-49bb-a609-653301fc203a" (UID: "ab785806-a92a-49bb-a609-653301fc203a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:07.703017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.702979 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" podUID="38865718-c2c2-48c3-aa91-47ce3a4fb022" containerName="main" containerID="cri-o://c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1" gracePeriod=45 Nov 01 16:29:07.714363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.714333 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkzqs\" (UniqueName: \"kubernetes.io/projected/ab785806-a92a-49bb-a609-653301fc203a-kube-api-access-rkzqs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:07.714363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.714361 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:07.714573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.714374 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8927f4c4-a345-48c1-bbb7-c231c39c4952-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:07.714573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.714382 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4xqd\" (UniqueName: \"kubernetes.io/projected/8927f4c4-a345-48c1-bbb7-c231c39c4952-kube-api-access-d4xqd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:07.714573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.714391 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab785806-a92a-49bb-a609-653301fc203a-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:07.962154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.962128 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:29:07.996967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.996940 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" Nov 01 16:29:07.997125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.996974 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222" event={"ID":"ab785806-a92a-49bb-a609-653301fc203a","Type":"ContainerDied","Data":"48e28ba5e2d0db6bb11b7d8f3f754b9f1e7148e11122bbf6e01a92fd9620f4cc"} Nov 01 16:29:07.997125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.997029 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48e28ba5e2d0db6bb11b7d8f3f754b9f1e7148e11122bbf6e01a92fd9620f4cc" Nov 01 16:29:07.999302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.999276 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" Nov 01 16:29:07.999453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.999280 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56" event={"ID":"8927f4c4-a345-48c1-bbb7-c231c39c4952","Type":"ContainerDied","Data":"ceeb7d424369130c117635f23fad732943b633a9de7166ae24bf8a312fba8cd9"} Nov 01 16:29:07.999453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:07.999403 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ceeb7d424369130c117635f23fad732943b633a9de7166ae24bf8a312fba8cd9" Nov 01 16:29:08.001174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.001135 4573 generic.go:356] "Generic (PLEG): container finished" podID="38865718-c2c2-48c3-aa91-47ce3a4fb022" containerID="c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1" exitCode=0 Nov 01 16:29:08.001289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.001204 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" event={"ID":"38865718-c2c2-48c3-aa91-47ce3a4fb022","Type":"ContainerDied","Data":"c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1"} Nov 01 16:29:08.001289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.001217 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" Nov 01 16:29:08.001289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.001234 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn" event={"ID":"38865718-c2c2-48c3-aa91-47ce3a4fb022","Type":"ContainerDied","Data":"0458f5978b52fb7a9f8f609b5bd656fb9ceb67601d8f234dffe1fc7791ce2ae8"} Nov 01 16:29:08.001289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.001258 4573 scope.go:122] "RemoveContainer" containerID="c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1" Nov 01 16:29:08.018689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.018634 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-model-cache\") pod \"38865718-c2c2-48c3-aa91-47ce3a4fb022\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " Nov 01 16:29:08.019315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.019288 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38865718-c2c2-48c3-aa91-47ce3a4fb022-tls-certs\" (UniqueName: \"kubernetes.io/secret/38865718-c2c2-48c3-aa91-47ce3a4fb022-tls-certs\") pod \"38865718-c2c2-48c3-aa91-47ce3a4fb022\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " Nov 01 16:29:08.019416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.019339 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-dshm\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-dshm\") pod \"38865718-c2c2-48c3-aa91-47ce3a4fb022\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " Nov 01 16:29:08.019416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.019411 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38865718-c2c2-48c3-aa91-47ce3a4fb022-kube-api-access-h98kg\" (UniqueName: \"kubernetes.io/projected/38865718-c2c2-48c3-aa91-47ce3a4fb022-kube-api-access-h98kg\") pod \"38865718-c2c2-48c3-aa91-47ce3a4fb022\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " Nov 01 16:29:08.019514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.019503 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-tmp-dir\") pod \"38865718-c2c2-48c3-aa91-47ce3a4fb022\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " Nov 01 16:29:08.019554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.019535 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-home\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-home\") pod \"38865718-c2c2-48c3-aa91-47ce3a4fb022\" (UID: \"38865718-c2c2-48c3-aa91-47ce3a4fb022\") " Nov 01 16:29:08.020579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.020144 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-model-cache" pod "38865718-c2c2-48c3-aa91-47ce3a4fb022" (UID: "38865718-c2c2-48c3-aa91-47ce3a4fb022"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:08.020579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.020282 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-tmp-dir" pod "38865718-c2c2-48c3-aa91-47ce3a4fb022" (UID: "38865718-c2c2-48c3-aa91-47ce3a4fb022"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:08.020579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.020299 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-home" pod "38865718-c2c2-48c3-aa91-47ce3a4fb022" (UID: "38865718-c2c2-48c3-aa91-47ce3a4fb022"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:08.023251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.023223 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38865718-c2c2-48c3-aa91-47ce3a4fb022-tls-certs" pod "38865718-c2c2-48c3-aa91-47ce3a4fb022" (UID: "38865718-c2c2-48c3-aa91-47ce3a4fb022"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:08.023359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.023280 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-dshm" pod "38865718-c2c2-48c3-aa91-47ce3a4fb022" (UID: "38865718-c2c2-48c3-aa91-47ce3a4fb022"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:08.023918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.023893 4573 scope.go:122] "RemoveContainer" containerID="c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1" Nov 01 16:29:08.024267 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:08.024243 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1\": container with ID starting with c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1 not found: ID does not exist" containerID="c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1" Nov 01 16:29:08.024360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.024280 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1"} err="failed to get container status \"c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1\": rpc error: code = NotFound desc = could not find container \"c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1\": container with ID starting with c591a1f01c349125430787eaf8244eede449704e8cc3992584811e333e9160a1 not found: ID does not exist" Nov 01 16:29:08.024936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.024913 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38865718-c2c2-48c3-aa91-47ce3a4fb022-kube-api-access-h98kg" pod "38865718-c2c2-48c3-aa91-47ce3a4fb022" (UID: "38865718-c2c2-48c3-aa91-47ce3a4fb022"). InnerVolumeSpecName "kube-api-access-h98kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:08.062471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.062432 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:08.121253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.121209 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:08.121253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.121241 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:08.121253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.121253 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:08.121508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.121267 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38865718-c2c2-48c3-aa91-47ce3a4fb022-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:08.121508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.121280 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38865718-c2c2-48c3-aa91-47ce3a4fb022-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:08.121508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.121293 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98kg\" (UniqueName: \"kubernetes.io/projected/38865718-c2c2-48c3-aa91-47ce3a4fb022-kube-api-access-h98kg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:08.383943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.383406 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn"] Nov 01 16:29:08.390377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.390355 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-734859/rate-test-model-734859-kserve-5b7d566745-wjgrn"] Nov 01 16:29:08.461615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.461579 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38865718-c2c2-48c3-aa91-47ce3a4fb022" path="/var/lib/kubelet/pods/38865718-c2c2-48c3-aa91-47ce3a4fb022/volumes" Nov 01 16:29:08.799830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.796942 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" containerID="cri-o://86e7944a47b83111cc1e7db4823b6d22fc3d154d83f6d41ce049ff8da9583a1b" gracePeriod=45 Nov 01 16:29:08.896542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:08.896498 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/ready\": dial tcp 10.133.0.53:8000: connect: connection refused" Nov 01 16:29:09.035805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.034886 4573 generic.go:356] "Generic (PLEG): container finished" podID="1438f4f8-2770-4839-bfe2-b2023337521e" containerID="86e7944a47b83111cc1e7db4823b6d22fc3d154d83f6d41ce049ff8da9583a1b" exitCode=0 Nov 01 16:29:09.035805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.034967 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" event={"ID":"1438f4f8-2770-4839-bfe2-b2023337521e","Type":"ContainerDied","Data":"86e7944a47b83111cc1e7db4823b6d22fc3d154d83f6d41ce049ff8da9583a1b"} Nov 01 16:29:09.192197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.191239 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:09.411782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.409456 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.448570 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1438f4f8-2770-4839-bfe2-b2023337521e-kube-api-access-hpfpj\" (UniqueName: \"kubernetes.io/projected/1438f4f8-2770-4839-bfe2-b2023337521e-kube-api-access-hpfpj\") pod \"1438f4f8-2770-4839-bfe2-b2023337521e\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.449079 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-model-cache\") pod \"1438f4f8-2770-4839-bfe2-b2023337521e\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.450392 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-model-cache" pod "1438f4f8-2770-4839-bfe2-b2023337521e" (UID: "1438f4f8-2770-4839-bfe2-b2023337521e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.450865 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1438f4f8-2770-4839-bfe2-b2023337521e-tls-certs\" (UniqueName: \"kubernetes.io/secret/1438f4f8-2770-4839-bfe2-b2023337521e-tls-certs\") pod \"1438f4f8-2770-4839-bfe2-b2023337521e\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.450972 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-home\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-home\") pod \"1438f4f8-2770-4839-bfe2-b2023337521e\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.451018 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-dshm\") pod \"1438f4f8-2770-4839-bfe2-b2023337521e\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.451064 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-tmp-dir\") pod \"1438f4f8-2770-4839-bfe2-b2023337521e\" (UID: \"1438f4f8-2770-4839-bfe2-b2023337521e\") " Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.451713 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.452842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.452186 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-home" pod "1438f4f8-2770-4839-bfe2-b2023337521e" (UID: "1438f4f8-2770-4839-bfe2-b2023337521e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.453480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.453391 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-tmp-dir" pod "1438f4f8-2770-4839-bfe2-b2023337521e" (UID: "1438f4f8-2770-4839-bfe2-b2023337521e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.454953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.454924 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1438f4f8-2770-4839-bfe2-b2023337521e-tls-certs" pod "1438f4f8-2770-4839-bfe2-b2023337521e" (UID: "1438f4f8-2770-4839-bfe2-b2023337521e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:09.455485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.455446 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1438f4f8-2770-4839-bfe2-b2023337521e-kube-api-access-hpfpj" pod "1438f4f8-2770-4839-bfe2-b2023337521e" (UID: "1438f4f8-2770-4839-bfe2-b2023337521e"). InnerVolumeSpecName "kube-api-access-hpfpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:09.456488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.456465 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-dshm" pod "1438f4f8-2770-4839-bfe2-b2023337521e" (UID: "1438f4f8-2770-4839-bfe2-b2023337521e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:09.553250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.553148 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1438f4f8-2770-4839-bfe2-b2023337521e-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.553250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.553183 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.553250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.553197 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.553250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.553208 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1438f4f8-2770-4839-bfe2-b2023337521e-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.553250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.553222 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpfpj\" (UniqueName: \"kubernetes.io/projected/1438f4f8-2770-4839-bfe2-b2023337521e-kube-api-access-hpfpj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:09.975320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:09.975289 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:10.081278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.081239 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" Nov 01 16:29:10.081476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.081282 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q" event={"ID":"1438f4f8-2770-4839-bfe2-b2023337521e","Type":"ContainerDied","Data":"ab5b76041805e1765c0085a857adf952b36ef7111099f13e2bd50617d8291a88"} Nov 01 16:29:10.081476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.081349 4573 scope.go:122] "RemoveContainer" containerID="86e7944a47b83111cc1e7db4823b6d22fc3d154d83f6d41ce049ff8da9583a1b" Nov 01 16:29:10.111558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.111523 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q"] Nov 01 16:29:10.120147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.120104 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-6b0e0f/simulator-w2-6b0e0f-kserve-6884f9f6d6-t5j6q"] Nov 01 16:29:10.460991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.460960 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" path="/var/lib/kubelet/pods/1438f4f8-2770-4839-bfe2-b2023337521e/volumes" Nov 01 16:29:10.792539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.792446 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:10.893117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.893091 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:29:10.893317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:10.893162 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:29:11.660176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.660140 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:11.666936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.666912 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6951-0e44db"] Nov 01 16:29:11.667641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667628 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8927f4c4-a345-48c1-bbb7-c231c39c4952" containerName="revoke-keys" Nov 01 16:29:11.667687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667643 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8927f4c4-a345-48c1-bbb7-c231c39c4952" containerName="revoke-keys" Nov 01 16:29:11.667687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667658 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ab785806-a92a-49bb-a609-653301fc203a" containerName="revoke-keys" Nov 01 16:29:11.667687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667663 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab785806-a92a-49bb-a609-653301fc203a" containerName="revoke-keys" Nov 01 16:29:11.667687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667677 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" Nov 01 16:29:11.667687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667682 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" Nov 01 16:29:11.667687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667689 4573 cpu_manager.go:414] "containerMap: removing container" podUID="38865718-c2c2-48c3-aa91-47ce3a4fb022" containerName="main" Nov 01 16:29:11.667945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667694 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38865718-c2c2-48c3-aa91-47ce3a4fb022" containerName="main" Nov 01 16:29:11.667945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667821 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab785806-a92a-49bb-a609-653301fc203a" containerName="revoke-keys" Nov 01 16:29:11.667945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667834 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1438f4f8-2770-4839-bfe2-b2023337521e" containerName="main" Nov 01 16:29:11.667945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667839 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="38865718-c2c2-48c3-aa91-47ce3a4fb022" containerName="main" Nov 01 16:29:11.667945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.667852 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8927f4c4-a345-48c1-bbb7-c231c39c4952" containerName="revoke-keys" Nov 01 16:29:11.672004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.671981 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6951-0e44db" Nov 01 16:29:11.673900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.673877 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6951-0e44db"] Nov 01 16:29:11.675051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.675021 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-hqwjk\"" Nov 01 16:29:11.699487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.699459 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xfgz\" (UniqueName: \"kubernetes.io/projected/b57e3cf4-c24f-4da3-9607-a3e70b048a66-kube-api-access-9xfgz\") pod \"test-curl-6951-0e44db\" (UID: \"b57e3cf4-c24f-4da3-9607-a3e70b048a66\") " pod="opendatahub/test-curl-6951-0e44db" Nov 01 16:29:11.800673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.800639 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xfgz\" (UniqueName: \"kubernetes.io/projected/b57e3cf4-c24f-4da3-9607-a3e70b048a66-kube-api-access-9xfgz\") pod \"test-curl-6951-0e44db\" (UID: \"b57e3cf4-c24f-4da3-9607-a3e70b048a66\") " pod="opendatahub/test-curl-6951-0e44db" Nov 01 16:29:11.807871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.807844 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xfgz\" (UniqueName: \"kubernetes.io/projected/b57e3cf4-c24f-4da3-9607-a3e70b048a66-kube-api-access-9xfgz\") pod \"test-curl-6951-0e44db\" (UID: \"b57e3cf4-c24f-4da3-9607-a3e70b048a66\") " pod="opendatahub/test-curl-6951-0e44db" Nov 01 16:29:11.988795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:11.988700 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6951-0e44db" Nov 01 16:29:12.269298 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:29:12.269262 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb57e3cf4_c24f_4da3_9607_a3e70b048a66.slice/crio-3a98c9d071c8fb29c00746342e09687177a27eb456bf59402a9bc0ad62e7333d WatchSource:0}: Error finding container 3a98c9d071c8fb29c00746342e09687177a27eb456bf59402a9bc0ad62e7333d: Status 404 returned error can't find the container with id 3a98c9d071c8fb29c00746342e09687177a27eb456bf59402a9bc0ad62e7333d Nov 01 16:29:12.269298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:12.269289 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6951-0e44db"] Nov 01 16:29:12.303617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:12.303587 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:12.975724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:12.975692 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:13.114015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:13.113981 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6951-0e44db" event={"ID":"b57e3cf4-c24f-4da3-9607-a3e70b048a66","Type":"ContainerStarted","Data":"3a98c9d071c8fb29c00746342e09687177a27eb456bf59402a9bc0ad62e7333d"} Nov 01 16:29:13.342955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:13.342869 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" podUID="915836fc-5724-4713-a688-dc226a3e7818" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:29:13.342955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:13.342890 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" podUID="46336b63-f773-4697-a916-8adf998d6da2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:29:13.766347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:13.766314 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:14.124424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.124385 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6951-0e44db" event={"ID":"b57e3cf4-c24f-4da3-9607-a3e70b048a66","Type":"ContainerStarted","Data":"a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444"} Nov 01 16:29:14.145593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.145534 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-6951-0e44db" podStartSLOduration=2.406392852 podStartE2EDuration="3.145515882s" totalImagesPullingTime="739.12303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:14.145138978 +0000 UTC m=+2491.848304909" watchObservedRunningTime="2026-11-01 16:29:14.145515882 +0000 UTC m=+2491.848681817" Nov 01 16:29:14.530388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.530299 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf"] Nov 01 16:29:14.535806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.535766 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.545618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.545583 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf"] Nov 01 16:29:14.547928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.547899 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3fd73f-serving-cert\"" Nov 01 16:29:14.569980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.569941 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:14.647588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.647555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:14.649099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.649074 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0118c8c5-5883-48ab-9457-ee52ec6b8f76-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.649360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.649334 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbqz\" (UniqueName: \"kubernetes.io/projected/0118c8c5-5883-48ab-9457-ee52ec6b8f76-kube-api-access-tlbqz\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.649422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.649404 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.649504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.649486 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.751191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.751155 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbqz\" (UniqueName: \"kubernetes.io/projected/0118c8c5-5883-48ab-9457-ee52ec6b8f76-kube-api-access-tlbqz\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.751416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.751271 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.751416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.751310 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.751416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.751367 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0118c8c5-5883-48ab-9457-ee52ec6b8f76-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.752196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.752170 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0118c8c5-5883-48ab-9457-ee52ec6b8f76-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.754138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.754113 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-metrics-certs\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.754346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.754322 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.759209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.759184 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbqz\" (UniqueName: \"kubernetes.io/projected/0118c8c5-5883-48ab-9457-ee52ec6b8f76-kube-api-access-tlbqz\") pod \"maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.806459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.806370 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6951-0e44db"] Nov 01 16:29:14.811637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.811599 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6951-0e44db"] Nov 01 16:29:14.865080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.865046 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:14.996173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.996134 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b"] Nov 01 16:29:14.996889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.996871 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b57e3cf4-c24f-4da3-9607-a3e70b048a66" containerName="test-curl-6951-0e44db" Nov 01 16:29:14.996889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.996888 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57e3cf4-c24f-4da3-9607-a3e70b048a66" containerName="test-curl-6951-0e44db" Nov 01 16:29:14.997086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:14.997011 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57e3cf4-c24f-4da3-9607-a3e70b048a66" containerName="test-curl-6951-0e44db" Nov 01 16:29:15.002063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.001999 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.006121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.005311 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:29:15.006121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.006087 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:29:15.013730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.011772 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b"] Nov 01 16:29:15.055589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.055538 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsqs5\" (UniqueName: \"kubernetes.io/projected/a9109b78-94ed-4e86-a286-2a488fd0f3d0-kube-api-access-wsqs5\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.055845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.055710 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.055958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.055909 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.150174 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:29:15.150144 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0118c8c5_5883_48ab_9457_ee52ec6b8f76.slice/crio-accef63858dc2742c0d91cb17f18b1e40c299c7aa551c3bc92e0f0cc29faaab2 WatchSource:0}: Error finding container accef63858dc2742c0d91cb17f18b1e40c299c7aa551c3bc92e0f0cc29faaab2: Status 404 returned error can't find the container with id accef63858dc2742c0d91cb17f18b1e40c299c7aa551c3bc92e0f0cc29faaab2 Nov 01 16:29:15.150535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.150177 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf"] Nov 01 16:29:15.157480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.157454 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.157571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.157561 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.157769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.157737 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsqs5\" (UniqueName: \"kubernetes.io/projected/a9109b78-94ed-4e86-a286-2a488fd0f3d0-kube-api-access-wsqs5\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.158152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.158122 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.160169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.160152 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.165068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.165048 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsqs5\" (UniqueName: \"kubernetes.io/projected/a9109b78-94ed-4e86-a286-2a488fd0f3d0-kube-api-access-wsqs5\") pod \"maas-api-revoke-sub-b806f5b8-gnk7b\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.330419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.330388 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:15.469875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.469838 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:15.622697 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:29:15.622666 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9109b78_94ed_4e86_a286_2a488fd0f3d0.slice/crio-8655b95b70c56edd81736747e74ac5cd36474156d00f2ad5c73c4a798c89bce5 WatchSource:0}: Error finding container 8655b95b70c56edd81736747e74ac5cd36474156d00f2ad5c73c4a798c89bce5: Status 404 returned error can't find the container with id 8655b95b70c56edd81736747e74ac5cd36474156d00f2ad5c73c4a798c89bce5 Nov 01 16:29:15.623139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:15.623116 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b"] Nov 01 16:29:16.142314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.142278 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" event={"ID":"0118c8c5-5883-48ab-9457-ee52ec6b8f76","Type":"ContainerStarted","Data":"7a21ce54b2f2ad119971f005f4dabdb3e9313d293d6f0e6bd863ea089e0b1422"} Nov 01 16:29:16.142501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.142344 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:16.142501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.142367 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" event={"ID":"0118c8c5-5883-48ab-9457-ee52ec6b8f76","Type":"ContainerStarted","Data":"accef63858dc2742c0d91cb17f18b1e40c299c7aa551c3bc92e0f0cc29faaab2"} Nov 01 16:29:16.144039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.144005 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" event={"ID":"a9109b78-94ed-4e86-a286-2a488fd0f3d0","Type":"ContainerStarted","Data":"1e07f1bb08eab1c52ffc2b28882ccdbe4dbb37d1b025edbb2f3ad0ac059a7d50"} Nov 01 16:29:16.144199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.144054 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" event={"ID":"a9109b78-94ed-4e86-a286-2a488fd0f3d0","Type":"ContainerStarted","Data":"8655b95b70c56edd81736747e74ac5cd36474156d00f2ad5c73c4a798c89bce5"} Nov 01 16:29:16.144276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.144187 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-6951-0e44db" podUID="b57e3cf4-c24f-4da3-9607-a3e70b048a66" containerName="test-curl-6951-0e44db" containerID="cri-o://a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444" gracePeriod=2 Nov 01 16:29:16.158903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.158862 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" podStartSLOduration=1.693779225 podStartE2EDuration="2.158848373s" totalImagesPullingTime="465.069148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:16.157786483 +0000 UTC m=+2493.860952421" watchObservedRunningTime="2026-11-01 16:29:16.158848373 +0000 UTC m=+2493.862014309" Nov 01 16:29:16.159876 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:16.159852 4573 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-6951-0e44db\" is forbidden: User \"system:node:ip-10-0-141-145.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-145.us-east-2.compute.internal' and this object" podUID="b57e3cf4-c24f-4da3-9607-a3e70b048a66" pod="opendatahub/test-curl-6951-0e44db" Nov 01 16:29:16.179149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.179107 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" podStartSLOduration=2.1790951769999998 podStartE2EDuration="2.179095177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:16.17790702 +0000 UTC m=+2493.881072969" watchObservedRunningTime="2026-11-01 16:29:16.179095177 +0000 UTC m=+2493.882261113" Nov 01 16:29:16.275478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:16.275407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:17.153718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:17.153684 4573 generic.go:356] "Generic (PLEG): container finished" podID="a9109b78-94ed-4e86-a286-2a488fd0f3d0" containerID="1e07f1bb08eab1c52ffc2b28882ccdbe4dbb37d1b025edbb2f3ad0ac059a7d50" exitCode=0 Nov 01 16:29:17.153923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:17.153777 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" event={"ID":"a9109b78-94ed-4e86-a286-2a488fd0f3d0","Type":"ContainerDied","Data":"1e07f1bb08eab1c52ffc2b28882ccdbe4dbb37d1b025edbb2f3ad0ac059a7d50"} Nov 01 16:29:17.171643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:17.171610 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:17.857910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:17.857870 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:17.960559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:17.960521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:18.340158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.340135 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:18.385246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.385216 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6951-0e44db" Nov 01 16:29:18.447235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.447162 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-openshift-ca\") pod \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " Nov 01 16:29:18.447235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.447210 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-xks-ca\") pod \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " Nov 01 16:29:18.447455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.447244 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9109b78-94ed-4e86-a286-2a488fd0f3d0-kube-api-access-wsqs5\" (UniqueName: \"kubernetes.io/projected/a9109b78-94ed-4e86-a286-2a488fd0f3d0-kube-api-access-wsqs5\") pod \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\" (UID: \"a9109b78-94ed-4e86-a286-2a488fd0f3d0\") " Nov 01 16:29:18.447513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.447464 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-openshift-ca" pod "a9109b78-94ed-4e86-a286-2a488fd0f3d0" (UID: "a9109b78-94ed-4e86-a286-2a488fd0f3d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:18.447889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.447864 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:18.449345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.449326 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-xks-ca" pod "a9109b78-94ed-4e86-a286-2a488fd0f3d0" (UID: "a9109b78-94ed-4e86-a286-2a488fd0f3d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:18.449455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.449404 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9109b78-94ed-4e86-a286-2a488fd0f3d0-kube-api-access-wsqs5" pod "a9109b78-94ed-4e86-a286-2a488fd0f3d0" (UID: "a9109b78-94ed-4e86-a286-2a488fd0f3d0"). InnerVolumeSpecName "kube-api-access-wsqs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:18.548725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.548694 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57e3cf4-c24f-4da3-9607-a3e70b048a66-kube-api-access-9xfgz\" (UniqueName: \"kubernetes.io/projected/b57e3cf4-c24f-4da3-9607-a3e70b048a66-kube-api-access-9xfgz\") pod \"b57e3cf4-c24f-4da3-9607-a3e70b048a66\" (UID: \"b57e3cf4-c24f-4da3-9607-a3e70b048a66\") " Nov 01 16:29:18.549981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.549915 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9109b78-94ed-4e86-a286-2a488fd0f3d0-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:18.549981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.549943 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsqs5\" (UniqueName: \"kubernetes.io/projected/a9109b78-94ed-4e86-a286-2a488fd0f3d0-kube-api-access-wsqs5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:18.550995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.550970 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57e3cf4-c24f-4da3-9607-a3e70b048a66-kube-api-access-9xfgz" pod "b57e3cf4-c24f-4da3-9607-a3e70b048a66" (UID: "b57e3cf4-c24f-4da3-9607-a3e70b048a66"). InnerVolumeSpecName "kube-api-access-9xfgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:18.619014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.618980 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:18.651961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:18.651931 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xfgz\" (UniqueName: \"kubernetes.io/projected/b57e3cf4-c24f-4da3-9607-a3e70b048a66-kube-api-access-9xfgz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:19.172534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.172506 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" Nov 01 16:29:19.172708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.172528 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b" event={"ID":"a9109b78-94ed-4e86-a286-2a488fd0f3d0","Type":"ContainerDied","Data":"8655b95b70c56edd81736747e74ac5cd36474156d00f2ad5c73c4a798c89bce5"} Nov 01 16:29:19.172708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.172579 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8655b95b70c56edd81736747e74ac5cd36474156d00f2ad5c73c4a798c89bce5" Nov 01 16:29:19.174277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.174249 4573 generic.go:356] "Generic (PLEG): container finished" podID="b57e3cf4-c24f-4da3-9607-a3e70b048a66" containerID="a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444" exitCode=137 Nov 01 16:29:19.174410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.174287 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6951-0e44db" Nov 01 16:29:19.174410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.174350 4573 scope.go:122] "RemoveContainer" containerID="a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444" Nov 01 16:29:19.187467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.187449 4573 scope.go:122] "RemoveContainer" containerID="a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444" Nov 01 16:29:19.187797 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:19.187778 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444\": container with ID starting with a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444 not found: ID does not exist" containerID="a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444" Nov 01 16:29:19.187860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.187806 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444"} err="failed to get container status \"a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444\": rpc error: code = NotFound desc = could not find container \"a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444\": container with ID starting with a594a8909874bec47f2b01c1744bc84b20bd8337b190215fdede008d2e3c5444 not found: ID does not exist" Nov 01 16:29:19.280368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.280331 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:19.964126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:19.964092 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:20.460082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:20.460046 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57e3cf4-c24f-4da3-9607-a3e70b048a66" path="/var/lib/kubelet/pods/b57e3cf4-c24f-4da3-9607-a3e70b048a66/volumes" Nov 01 16:29:20.626053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:20.626018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:21.251157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:21.251122 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:21.964118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:21.964081 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:22.159674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:22.159642 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:29:22.865087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:22.865048 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:23.343099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:23.342996 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" podUID="915836fc-5724-4713-a688-dc226a3e7818" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:29:23.343099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:23.343008 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" podUID="46336b63-f773-4697-a916-8adf998d6da2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:29:23.526610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:23.526568 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:24.038904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.038872 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg"] Nov 01 16:29:24.039546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.039508 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" podUID="c7a08917-2376-41da-b0bb-f814b4790e32" containerName="maas-api" containerID="cri-o://315e0306e2020f0ecf028c9c08e98f0e018ed1080d68482fdd53f0ae9559d313" gracePeriod=30 Nov 01 16:29:24.057794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.057409 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:24.160154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.160019 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:24.250937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.250259 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:24.256781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.255043 4573 generic.go:356] "Generic (PLEG): container finished" podID="c7a08917-2376-41da-b0bb-f814b4790e32" containerID="315e0306e2020f0ecf028c9c08e98f0e018ed1080d68482fdd53f0ae9559d313" exitCode=0 Nov 01 16:29:24.256781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.255133 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" event={"ID":"c7a08917-2376-41da-b0bb-f814b4790e32","Type":"ContainerDied","Data":"315e0306e2020f0ecf028c9c08e98f0e018ed1080d68482fdd53f0ae9559d313"} Nov 01 16:29:24.331916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.331857 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:24.370233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.370206 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs"] Nov 01 16:29:24.372963 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:24.372936 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:29:24.374472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.374452 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-7321d6-nsjzs"] Nov 01 16:29:24.429578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.429549 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:24.488775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.480769 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d22278b-8fe9-401d-bbf6-fe96c5b443c8" path="/var/lib/kubelet/pods/2d22278b-8fe9-401d-bbf6-fe96c5b443c8/volumes" Nov 01 16:29:24.501774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.497327 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:29:24.596026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.595072 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:24.596026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.595863 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls\") pod \"c7a08917-2376-41da-b0bb-f814b4790e32\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " Nov 01 16:29:24.596026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.595935 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-metrics-certs\") pod \"c7a08917-2376-41da-b0bb-f814b4790e32\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " Nov 01 16:29:24.596367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.596105 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a08917-2376-41da-b0bb-f814b4790e32-kube-api-access-5zb9t\" (UniqueName: \"kubernetes.io/projected/c7a08917-2376-41da-b0bb-f814b4790e32-kube-api-access-5zb9t\") pod \"c7a08917-2376-41da-b0bb-f814b4790e32\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " Nov 01 16:29:24.596367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.596213 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a08917-2376-41da-b0bb-f814b4790e32-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a08917-2376-41da-b0bb-f814b4790e32-openshift-service-ca\") pod \"c7a08917-2376-41da-b0bb-f814b4790e32\" (UID: \"c7a08917-2376-41da-b0bb-f814b4790e32\") " Nov 01 16:29:24.624817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.624461 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a08917-2376-41da-b0bb-f814b4790e32-openshift-service-ca" pod "c7a08917-2376-41da-b0bb-f814b4790e32" (UID: "c7a08917-2376-41da-b0bb-f814b4790e32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:24.628827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.626186 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-metrics-certs" pod "c7a08917-2376-41da-b0bb-f814b4790e32" (UID: "c7a08917-2376-41da-b0bb-f814b4790e32"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:24.695495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.695432 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls" pod "c7a08917-2376-41da-b0bb-f814b4790e32" (UID: "c7a08917-2376-41da-b0bb-f814b4790e32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:24.700798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.698572 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a08917-2376-41da-b0bb-f814b4790e32-kube-api-access-5zb9t" pod "c7a08917-2376-41da-b0bb-f814b4790e32" (UID: "c7a08917-2376-41da-b0bb-f814b4790e32"). InnerVolumeSpecName "kube-api-access-5zb9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:24.700798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.699108 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zb9t\" (UniqueName: \"kubernetes.io/projected/c7a08917-2376-41da-b0bb-f814b4790e32-kube-api-access-5zb9t\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:24.700798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.699128 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7a08917-2376-41da-b0bb-f814b4790e32-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:24.700798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.699139 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:24.700798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.699149 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7a08917-2376-41da-b0bb-f814b4790e32-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:24.747564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.747534 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 16:29:24.801797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.800863 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/915836fc-5724-4713-a688-dc226a3e7818-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915836fc-5724-4713-a688-dc226a3e7818-plugins-config-volume\") pod \"915836fc-5724-4713-a688-dc226a3e7818\" (UID: \"915836fc-5724-4713-a688-dc226a3e7818\") " Nov 01 16:29:24.801797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.800957 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915836fc-5724-4713-a688-dc226a3e7818-kube-api-access-srf64\" (UniqueName: \"kubernetes.io/projected/915836fc-5724-4713-a688-dc226a3e7818-kube-api-access-srf64\") pod \"915836fc-5724-4713-a688-dc226a3e7818\" (UID: \"915836fc-5724-4713-a688-dc226a3e7818\") " Nov 01 16:29:24.802784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.802724 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/915836fc-5724-4713-a688-dc226a3e7818-plugins-config-volume" pod "915836fc-5724-4713-a688-dc226a3e7818" (UID: "915836fc-5724-4713-a688-dc226a3e7818"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:24.804640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.804605 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915836fc-5724-4713-a688-dc226a3e7818-kube-api-access-srf64" pod "915836fc-5724-4713-a688-dc226a3e7818" (UID: "915836fc-5724-4713-a688-dc226a3e7818"). InnerVolumeSpecName "kube-api-access-srf64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:24.805992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.805965 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:24.859787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.859746 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 16:29:24.903237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.903199 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46336b63-f773-4697-a916-8adf998d6da2-kube-api-access-hdfqg\" (UniqueName: \"kubernetes.io/projected/46336b63-f773-4697-a916-8adf998d6da2-kube-api-access-hdfqg\") pod \"46336b63-f773-4697-a916-8adf998d6da2\" (UID: \"46336b63-f773-4697-a916-8adf998d6da2\") " Nov 01 16:29:24.903433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.903328 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46336b63-f773-4697-a916-8adf998d6da2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46336b63-f773-4697-a916-8adf998d6da2-plugins-config-volume\") pod \"46336b63-f773-4697-a916-8adf998d6da2\" (UID: \"46336b63-f773-4697-a916-8adf998d6da2\") " Nov 01 16:29:24.903924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.903894 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46336b63-f773-4697-a916-8adf998d6da2-plugins-config-volume" pod "46336b63-f773-4697-a916-8adf998d6da2" (UID: "46336b63-f773-4697-a916-8adf998d6da2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:24.904056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.903930 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/915836fc-5724-4713-a688-dc226a3e7818-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:24.904056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.903952 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srf64\" (UniqueName: \"kubernetes.io/projected/915836fc-5724-4713-a688-dc226a3e7818-kube-api-access-srf64\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:24.906444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.906414 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46336b63-f773-4697-a916-8adf998d6da2-kube-api-access-hdfqg" pod "46336b63-f773-4697-a916-8adf998d6da2" (UID: "46336b63-f773-4697-a916-8adf998d6da2"). InnerVolumeSpecName "kube-api-access-hdfqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:24.920666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:24.912032 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:25.005305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.005274 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdfqg\" (UniqueName: \"kubernetes.io/projected/46336b63-f773-4697-a916-8adf998d6da2-kube-api-access-hdfqg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.005305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.005301 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46336b63-f773-4697-a916-8adf998d6da2-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:25.091512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.091476 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:25.268179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.268142 4573 generic.go:356] "Generic (PLEG): container finished" podID="915836fc-5724-4713-a688-dc226a3e7818" containerID="7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf" exitCode=0 Nov 01 16:29:25.268341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.268304 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" event={"ID":"915836fc-5724-4713-a688-dc226a3e7818","Type":"ContainerDied","Data":"7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf"} Nov 01 16:29:25.268435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.268337 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" Nov 01 16:29:25.268435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.268370 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh" event={"ID":"915836fc-5724-4713-a688-dc226a3e7818","Type":"ContainerDied","Data":"031ac16301c435519f34626a9a557b1b8887eea83ad4becd8b04462b46822b88"} Nov 01 16:29:25.268435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.268413 4573 scope.go:122] "RemoveContainer" containerID="7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf" Nov 01 16:29:25.270018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.269971 4573 generic.go:356] "Generic (PLEG): container finished" podID="46336b63-f773-4697-a916-8adf998d6da2" containerID="f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9" exitCode=0 Nov 01 16:29:25.270159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.270145 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" Nov 01 16:29:25.270472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.270449 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" event={"ID":"46336b63-f773-4697-a916-8adf998d6da2","Type":"ContainerDied","Data":"f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9"} Nov 01 16:29:25.270563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.270534 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p" event={"ID":"46336b63-f773-4697-a916-8adf998d6da2","Type":"ContainerDied","Data":"32a3a45446f81d5d205452a0006715743041f0c8aea14726ae4268b6be97bf9d"} Nov 01 16:29:25.272372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.272318 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" event={"ID":"c7a08917-2376-41da-b0bb-f814b4790e32","Type":"ContainerDied","Data":"827e60c602db2283ac3eca61a2ad52fd457830388081cf3d630744b7ecd753e7"} Nov 01 16:29:25.272521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.272397 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg" Nov 01 16:29:25.283431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.283410 4573 scope.go:122] "RemoveContainer" containerID="7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf" Nov 01 16:29:25.283844 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:25.283821 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf\": container with ID starting with 7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf not found: ID does not exist" containerID="7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf" Nov 01 16:29:25.283961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.283853 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf"} err="failed to get container status \"7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf\": rpc error: code = NotFound desc = could not find container \"7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf\": container with ID starting with 7ef22940331388584ab559a8e9a738d330c6c120f9389a976093e4c579fcdbcf not found: ID does not exist" Nov 01 16:29:25.283961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.283876 4573 scope.go:122] "RemoveContainer" containerID="f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9" Nov 01 16:29:25.294579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.294563 4573 scope.go:122] "RemoveContainer" containerID="f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9" Nov 01 16:29:25.294883 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:25.294864 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9\": container with ID starting with f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9 not found: ID does not exist" containerID="f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9" Nov 01 16:29:25.294967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.294886 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9"} err="failed to get container status \"f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9\": rpc error: code = NotFound desc = could not find container \"f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9\": container with ID starting with f21ca6c2c01e438d104b5c87518eb1457f13eb0c5c51d321e5b2d2b24998a3f9 not found: ID does not exist" Nov 01 16:29:25.294967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.294902 4573 scope.go:122] "RemoveContainer" containerID="315e0306e2020f0ecf028c9c08e98f0e018ed1080d68482fdd53f0ae9559d313" Nov 01 16:29:25.308576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.308552 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p"] Nov 01 16:29:25.319325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.319302 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-6b0e0f-765874f899-9752p"] Nov 01 16:29:25.331816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.331783 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh"] Nov 01 16:29:25.345254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.345222 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-6b0e0f-7bc6c99cbf-scggh"] Nov 01 16:29:25.354102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.354069 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg"] Nov 01 16:29:25.360734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.360710 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7321d6-865db7474b-5khgg"] Nov 01 16:29:25.384099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.384071 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:25.975194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:25.975157 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:26.566773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:26.566207 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46336b63-f773-4697-a916-8adf998d6da2" path="/var/lib/kubelet/pods/46336b63-f773-4697-a916-8adf998d6da2/volumes" Nov 01 16:29:26.576773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:26.570271 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915836fc-5724-4713-a688-dc226a3e7818" path="/var/lib/kubelet/pods/915836fc-5724-4713-a688-dc226a3e7818/volumes" Nov 01 16:29:26.576773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:26.570853 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a08917-2376-41da-b0bb-f814b4790e32" path="/var/lib/kubelet/pods/c7a08917-2376-41da-b0bb-f814b4790e32/volumes" Nov 01 16:29:26.943610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:26.943577 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:27.688297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:27.688261 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:28.375810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:28.375777 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:29.067837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:29.067795 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:29.687492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:29.687459 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:30.456079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:30.456042 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:31.343200 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:31.343172 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:31.701469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:31.701433 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:31.902389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:31.902350 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:32.001452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:32.001380 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:32.132514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:32.132476 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:32.226898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:32.226851 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:32.325414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:32.325329 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:32.624665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:32.624634 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:32.741774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:32.740946 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:32.929225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:32.929134 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58"] Nov 01 16:29:32.936824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:32.936795 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl"] Nov 01 16:29:33.008390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.008354 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:33.100670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.100632 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:33.268440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.266320 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:33.495287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.495254 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:33.598171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.598086 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:33.771987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.771948 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:33.796517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.796486 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq"] Nov 01 16:29:33.796782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.796736 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" containerID="cri-o://bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9" gracePeriod=30 Nov 01 16:29:33.877668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:33.877635 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:34.547932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:34.547893 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:35.190402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:35.190371 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:36.010015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:36.009964 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:37.026670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:37.026622 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:37.265708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:37.265676 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79"] Nov 01 16:29:37.293432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:37.293354 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:37.929824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:37.929777 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" podUID="bf69a1d6-83d6-40f9-b798-5502d0b380e0" containerName="payload-processing" containerID="cri-o://3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2" gracePeriod=25 Nov 01 16:29:37.937339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:37.937309 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" podUID="86149c0b-b1a1-4b7b-a618-eb47b706771d" containerName="payload-pre-processing" containerID="cri-o://242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3" gracePeriod=25 Nov 01 16:29:37.965512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:37.965487 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:38.298240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.298209 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:29:38.391804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.391779 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:29:38.430376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.430336 4573 generic.go:356] "Generic (PLEG): container finished" podID="86149c0b-b1a1-4b7b-a618-eb47b706771d" containerID="242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3" exitCode=0 Nov 01 16:29:38.430545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.430391 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" Nov 01 16:29:38.430545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.430415 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" event={"ID":"86149c0b-b1a1-4b7b-a618-eb47b706771d","Type":"ContainerDied","Data":"242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3"} Nov 01 16:29:38.430545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.430505 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl" event={"ID":"86149c0b-b1a1-4b7b-a618-eb47b706771d","Type":"ContainerDied","Data":"60a6d2c357eb81912f167bc8bd058b16a624908850021d5d8cef8e031b99d89e"} Nov 01 16:29:38.430545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.430535 4573 scope.go:122] "RemoveContainer" containerID="242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3" Nov 01 16:29:38.433175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.433151 4573 generic.go:356] "Generic (PLEG): container finished" podID="bf69a1d6-83d6-40f9-b798-5502d0b380e0" containerID="3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2" exitCode=0 Nov 01 16:29:38.433310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.433184 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" event={"ID":"bf69a1d6-83d6-40f9-b798-5502d0b380e0","Type":"ContainerDied","Data":"3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2"} Nov 01 16:29:38.433310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.433202 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" Nov 01 16:29:38.433310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.433219 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58" event={"ID":"bf69a1d6-83d6-40f9-b798-5502d0b380e0","Type":"ContainerDied","Data":"efd3d379c03b426356fae844bcff92e7e9d6ae2389a53ea7e312189609fa15a9"} Nov 01 16:29:38.449368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.449340 4573 scope.go:122] "RemoveContainer" containerID="242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3" Nov 01 16:29:38.451876 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:38.451829 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3\": container with ID starting with 242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3 not found: ID does not exist" containerID="242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3" Nov 01 16:29:38.452012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.451870 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3"} err="failed to get container status \"242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3\": rpc error: code = NotFound desc = could not find container \"242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3\": container with ID starting with 242420b41c43d038c46eadf66c42fdba01a3328e1f1dbe3717fd68edbb5da7b3 not found: ID does not exist" Nov 01 16:29:38.452012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.451894 4573 scope.go:122] "RemoveContainer" containerID="3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2" Nov 01 16:29:38.457360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.457334 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:38.459370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.459344 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf69a1d6-83d6-40f9-b798-5502d0b380e0-kube-api-access-fc8mb\" (UniqueName: \"kubernetes.io/projected/bf69a1d6-83d6-40f9-b798-5502d0b380e0-kube-api-access-fc8mb\") pod \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\" (UID: \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\") " Nov 01 16:29:38.459456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.459427 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf69a1d6-83d6-40f9-b798-5502d0b380e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf69a1d6-83d6-40f9-b798-5502d0b380e0-plugins-config-volume\") pod \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\" (UID: \"bf69a1d6-83d6-40f9-b798-5502d0b380e0\") " Nov 01 16:29:38.460363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.460337 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf69a1d6-83d6-40f9-b798-5502d0b380e0-plugins-config-volume" pod "bf69a1d6-83d6-40f9-b798-5502d0b380e0" (UID: "bf69a1d6-83d6-40f9-b798-5502d0b380e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:38.462382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.462354 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf69a1d6-83d6-40f9-b798-5502d0b380e0-kube-api-access-fc8mb" pod "bf69a1d6-83d6-40f9-b798-5502d0b380e0" (UID: "bf69a1d6-83d6-40f9-b798-5502d0b380e0"). InnerVolumeSpecName "kube-api-access-fc8mb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:38.467608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.467574 4573 scope.go:122] "RemoveContainer" containerID="3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2" Nov 01 16:29:38.467933 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:38.467910 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2\": container with ID starting with 3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2 not found: ID does not exist" containerID="3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2" Nov 01 16:29:38.468026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.467946 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2"} err="failed to get container status \"3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2\": rpc error: code = NotFound desc = could not find container \"3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2\": container with ID starting with 3f547d042cc5c2f822a053295e2b9d93f9e44bdef5914928009a055c4108c0f2 not found: ID does not exist" Nov 01 16:29:38.561103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.561079 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86149c0b-b1a1-4b7b-a618-eb47b706771d-kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/86149c0b-b1a1-4b7b-a618-eb47b706771d-kube-api-access-8b4ws\") pod \"86149c0b-b1a1-4b7b-a618-eb47b706771d\" (UID: \"86149c0b-b1a1-4b7b-a618-eb47b706771d\") " Nov 01 16:29:38.561258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.561127 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86149c0b-b1a1-4b7b-a618-eb47b706771d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86149c0b-b1a1-4b7b-a618-eb47b706771d-plugins-config-volume\") pod \"86149c0b-b1a1-4b7b-a618-eb47b706771d\" (UID: \"86149c0b-b1a1-4b7b-a618-eb47b706771d\") " Nov 01 16:29:38.561494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.561472 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86149c0b-b1a1-4b7b-a618-eb47b706771d-plugins-config-volume" pod "86149c0b-b1a1-4b7b-a618-eb47b706771d" (UID: "86149c0b-b1a1-4b7b-a618-eb47b706771d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:38.562310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.562285 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86149c0b-b1a1-4b7b-a618-eb47b706771d-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:38.562406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.562314 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc8mb\" (UniqueName: \"kubernetes.io/projected/bf69a1d6-83d6-40f9-b798-5502d0b380e0-kube-api-access-fc8mb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:38.562406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.562331 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf69a1d6-83d6-40f9-b798-5502d0b380e0-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:38.563412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.563392 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86149c0b-b1a1-4b7b-a618-eb47b706771d-kube-api-access-8b4ws" pod "86149c0b-b1a1-4b7b-a618-eb47b706771d" (UID: "86149c0b-b1a1-4b7b-a618-eb47b706771d"). InnerVolumeSpecName "kube-api-access-8b4ws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:38.663624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.663597 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b4ws\" (UniqueName: \"kubernetes.io/projected/86149c0b-b1a1-4b7b-a618-eb47b706771d-kube-api-access-8b4ws\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:38.734329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.734254 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:38.756129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.755647 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58"] Nov 01 16:29:38.761785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.761744 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3fd73f-596f898c95-xsw58"] Nov 01 16:29:38.770032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.769998 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl"] Nov 01 16:29:38.775863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:38.775841 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-745bf87d99-rh4hl"] Nov 01 16:29:39.084295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.084269 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:29:39.169804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.169768 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istiod-ca-cert\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.169835 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-socket\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.169861 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-certs\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.169891 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-kube-api-access-zn8d9\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-kube-api-access-zn8d9\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170063 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-podinfo\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170101 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-envoy\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170186 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-socket" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.170218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170198 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istiod-ca-cert" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:29:39.170436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170204 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-certs" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.170436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170207 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-credential-socket\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170300 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-data\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170411 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-credential-socket" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.170628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170449 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-token\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-token\") pod \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\" (UID: \"c40bde30-47d6-4343-9ba4-780dab5f1ce8\") " Nov 01 16:29:39.170628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.170596 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-data" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.171239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.171216 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.171354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.171242 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.171354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.171256 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.171354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.171268 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.171354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.171281 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.173536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.173021 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-podinfo" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:29:39.173536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.173487 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-kube-api-access-zn8d9" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "kube-api-access-zn8d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:39.173536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.173514 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-token" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:39.173883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.173858 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-envoy" pod "c40bde30-47d6-4343-9ba4-780dab5f1ce8" (UID: "c40bde30-47d6-4343-9ba4-780dab5f1ce8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:39.273122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.273094 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.273122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.273120 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.273285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.273129 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.273285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.273139 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8d9\" (UniqueName: \"kubernetes.io/projected/c40bde30-47d6-4343-9ba4-780dab5f1ce8-kube-api-access-zn8d9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:39.442289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.442256 4573 generic.go:356] "Generic (PLEG): container finished" podID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerID="bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9" exitCode=0 Nov 01 16:29:39.442870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.442303 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" Nov 01 16:29:39.442870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.442338 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" event={"ID":"c40bde30-47d6-4343-9ba4-780dab5f1ce8","Type":"ContainerDied","Data":"bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9"} Nov 01 16:29:39.442870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.442395 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq" event={"ID":"c40bde30-47d6-4343-9ba4-780dab5f1ce8","Type":"ContainerDied","Data":"981ac3db570f216ffe5935e0881402f1b00cd363d253a93f6eb39759a64b9035"} Nov 01 16:29:39.442870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.442424 4573 scope.go:122] "RemoveContainer" containerID="bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9" Nov 01 16:29:39.455503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.455487 4573 scope.go:122] "RemoveContainer" containerID="bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9" Nov 01 16:29:39.455848 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:39.455821 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9\": container with ID starting with bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9 not found: ID does not exist" containerID="bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9" Nov 01 16:29:39.455930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.455860 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9"} err="failed to get container status \"bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9\": rpc error: code = NotFound desc = could not find container \"bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9\": container with ID starting with bbd5b34efd7b5dd90ea10f4e5257dbf0e7aefe0ba068c45a9a497f8cf065bba9 not found: ID does not exist" Nov 01 16:29:39.457659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.457631 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:39.467943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.467921 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq"] Nov 01 16:29:39.475616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:39.475590 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-6b0e0f-openshift-default-7966746dbc-65hkq"] Nov 01 16:29:40.343803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:40.343738 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:40.465004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:40.464966 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86149c0b-b1a1-4b7b-a618-eb47b706771d" path="/var/lib/kubelet/pods/86149c0b-b1a1-4b7b-a618-eb47b706771d/volumes" Nov 01 16:29:40.465657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:40.465638 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf69a1d6-83d6-40f9-b798-5502d0b380e0" path="/var/lib/kubelet/pods/bf69a1d6-83d6-40f9-b798-5502d0b380e0/volumes" Nov 01 16:29:40.466205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:40.466191 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" path="/var/lib/kubelet/pods/c40bde30-47d6-4343-9ba4-780dab5f1ce8/volumes" Nov 01 16:29:41.217270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:41.217235 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:41.983583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:41.983550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:42.793688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:42.793648 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:43.144965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:43.144924 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:43.372591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:43.372550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:43.470201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:43.470117 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:43.718533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:43.718418 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:43.820736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:43.820642 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:44.250139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:44.250102 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:44.543437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:44.543363 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:45.026372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:45.026336 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:45.752727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:45.752686 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:46.467637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:46.467605 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:47.437376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:47.437339 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:47.959260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:47.959228 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:48.065712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:48.065656 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:48.395044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:48.395016 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:48.899600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:48.899563 4573 scope.go:122] "RemoveContainer" containerID="3a70d85f21843463b7e9d9629e2920790e47746b65216fa605dcc7776c620a51" Nov 01 16:29:48.913982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:48.913950 4573 scope.go:122] "RemoveContainer" containerID="57ee74ba3924b50682101b5038b8406e519e27c47bf6532712f2d63c426ac966" Nov 01 16:29:48.923780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:48.923730 4573 scope.go:122] "RemoveContainer" containerID="ca61cfd01af812ef6b13950b59907584c67d54d17fee877442e783d3afcbd260" Nov 01 16:29:48.942561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:48.942532 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:49.042899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.042866 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:49.043727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.043708 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw"] Nov 01 16:29:49.044405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044390 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bf69a1d6-83d6-40f9-b798-5502d0b380e0" containerName="payload-processing" Nov 01 16:29:49.044450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044407 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf69a1d6-83d6-40f9-b798-5502d0b380e0" containerName="payload-processing" Nov 01 16:29:49.044450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044421 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a9109b78-94ed-4e86-a286-2a488fd0f3d0" containerName="revoke-keys" Nov 01 16:29:49.044450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044426 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9109b78-94ed-4e86-a286-2a488fd0f3d0" containerName="revoke-keys" Nov 01 16:29:49.044450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044435 4573 cpu_manager.go:414] "containerMap: removing container" podUID="86149c0b-b1a1-4b7b-a618-eb47b706771d" containerName="payload-pre-processing" Nov 01 16:29:49.044450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044440 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86149c0b-b1a1-4b7b-a618-eb47b706771d" containerName="payload-pre-processing" Nov 01 16:29:49.044450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044451 4573 cpu_manager.go:414] "containerMap: removing container" podUID="46336b63-f773-4697-a916-8adf998d6da2" containerName="payload-pre-processing" Nov 01 16:29:49.044650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044456 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46336b63-f773-4697-a916-8adf998d6da2" containerName="payload-pre-processing" Nov 01 16:29:49.044650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044507 4573 cpu_manager.go:414] "containerMap: removing container" podUID="915836fc-5724-4713-a688-dc226a3e7818" containerName="payload-processing" Nov 01 16:29:49.044650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044513 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915836fc-5724-4713-a688-dc226a3e7818" containerName="payload-processing" Nov 01 16:29:49.044650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044521 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c7a08917-2376-41da-b0bb-f814b4790e32" containerName="maas-api" Nov 01 16:29:49.044650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044527 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a08917-2376-41da-b0bb-f814b4790e32" containerName="maas-api" Nov 01 16:29:49.044650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044535 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" Nov 01 16:29:49.044650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044540 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" Nov 01 16:29:49.045013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044657 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="86149c0b-b1a1-4b7b-a618-eb47b706771d" containerName="payload-pre-processing" Nov 01 16:29:49.045013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044667 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="46336b63-f773-4697-a916-8adf998d6da2" containerName="payload-pre-processing" Nov 01 16:29:49.045013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044678 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="915836fc-5724-4713-a688-dc226a3e7818" containerName="payload-processing" Nov 01 16:29:49.045013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044684 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a08917-2376-41da-b0bb-f814b4790e32" containerName="maas-api" Nov 01 16:29:49.045013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044690 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40bde30-47d6-4343-9ba4-780dab5f1ce8" containerName="istio-proxy" Nov 01 16:29:49.045013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044700 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf69a1d6-83d6-40f9-b798-5502d0b380e0" containerName="payload-processing" Nov 01 16:29:49.045013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.044705 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9109b78-94ed-4e86-a286-2a488fd0f3d0" containerName="revoke-keys" Nov 01 16:29:49.051205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.051185 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:49.060348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.060300 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw"] Nov 01 16:29:49.060625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.060600 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3fd73f-dockercfg-n5xdq\"" Nov 01 16:29:49.061307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.060700 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3fd73f\"" Nov 01 16:29:49.127091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.127058 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753ad89-a39d-428b-ab90-c2923fd61ad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw\" (UID: \"2753ad89-a39d-428b-ab90-c2923fd61ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:49.127288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.127241 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fplrj\" (UniqueName: \"kubernetes.io/projected/2753ad89-a39d-428b-ab90-c2923fd61ad9-kube-api-access-fplrj\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw\" (UID: \"2753ad89-a39d-428b-ab90-c2923fd61ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:49.207541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.207459 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:49.228310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.228280 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753ad89-a39d-428b-ab90-c2923fd61ad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw\" (UID: \"2753ad89-a39d-428b-ab90-c2923fd61ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:49.228469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.228390 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fplrj\" (UniqueName: \"kubernetes.io/projected/2753ad89-a39d-428b-ab90-c2923fd61ad9-kube-api-access-fplrj\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw\" (UID: \"2753ad89-a39d-428b-ab90-c2923fd61ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:49.229220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.229192 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753ad89-a39d-428b-ab90-c2923fd61ad9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw\" (UID: \"2753ad89-a39d-428b-ab90-c2923fd61ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:49.237865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.237840 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv"] Nov 01 16:29:49.245245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.245225 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:49.250388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.250355 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fplrj\" (UniqueName: \"kubernetes.io/projected/2753ad89-a39d-428b-ab90-c2923fd61ad9-kube-api-access-fplrj\") pod \"payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw\" (UID: \"2753ad89-a39d-428b-ab90-c2923fd61ad9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:49.256491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.256468 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv"] Nov 01 16:29:49.259917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.259895 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-028c46\"" Nov 01 16:29:49.305267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.305234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:49.313045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.313012 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c"] Nov 01 16:29:49.317431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.317410 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:49.330404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.330368 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-028c46-dockercfg-8kfjt\"" Nov 01 16:29:49.330926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.330890 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv4ts\" (UniqueName: \"kubernetes.io/projected/7e74a675-d484-496a-a223-1d03ca77f424-kube-api-access-nv4ts\") pod \"payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv\" (UID: \"7e74a675-d484-496a-a223-1d03ca77f424\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:49.331047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.331025 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74a675-d484-496a-a223-1d03ca77f424-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv\" (UID: \"7e74a675-d484-496a-a223-1d03ca77f424\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:49.333906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.333885 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c"] Nov 01 16:29:49.384263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.384219 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:49.477003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.446918 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv4ts\" (UniqueName: \"kubernetes.io/projected/7e74a675-d484-496a-a223-1d03ca77f424-kube-api-access-nv4ts\") pod \"payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv\" (UID: \"7e74a675-d484-496a-a223-1d03ca77f424\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:49.477003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.447057 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74a675-d484-496a-a223-1d03ca77f424-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv\" (UID: \"7e74a675-d484-496a-a223-1d03ca77f424\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:49.477003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.448085 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74a675-d484-496a-a223-1d03ca77f424-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv\" (UID: \"7e74a675-d484-496a-a223-1d03ca77f424\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:49.507506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.507433 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv4ts\" (UniqueName: \"kubernetes.io/projected/7e74a675-d484-496a-a223-1d03ca77f424-kube-api-access-nv4ts\") pod \"payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv\" (UID: \"7e74a675-d484-496a-a223-1d03ca77f424\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:49.550936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.549687 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7kp\" (UniqueName: \"kubernetes.io/projected/839d2919-d17c-4e19-bdba-dcbb222ba1d1-kube-api-access-zr7kp\") pod \"payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c\" (UID: \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:49.553662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.553418 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839d2919-d17c-4e19-bdba-dcbb222ba1d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c\" (UID: \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:49.592782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.586344 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:49.661642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.661602 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:49.663618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.663204 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7kp\" (UniqueName: \"kubernetes.io/projected/839d2919-d17c-4e19-bdba-dcbb222ba1d1-kube-api-access-zr7kp\") pod \"payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c\" (UID: \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:49.664666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.663736 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839d2919-d17c-4e19-bdba-dcbb222ba1d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c\" (UID: \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:49.664836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.664632 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839d2919-d17c-4e19-bdba-dcbb222ba1d1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c\" (UID: \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:49.671518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.671494 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7kp\" (UniqueName: \"kubernetes.io/projected/839d2919-d17c-4e19-bdba-dcbb222ba1d1-kube-api-access-zr7kp\") pod \"payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c\" (UID: \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:49.853095 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:29:49.853064 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2753ad89_a39d_428b_ab90_c2923fd61ad9.slice/crio-b96d19df608ad54429b74d0a726ec0a454f973bbd19135bebcd5a385a8cd5d42 WatchSource:0}: Error finding container b96d19df608ad54429b74d0a726ec0a454f973bbd19135bebcd5a385a8cd5d42: Status 404 returned error can't find the container with id b96d19df608ad54429b74d0a726ec0a454f973bbd19135bebcd5a385a8cd5d42 Nov 01 16:29:49.853312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.853289 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw"] Nov 01 16:29:49.923180 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:29:49.923157 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e74a675_d484_496a_a223_1d03ca77f424.slice/crio-affe450ee2433e07d45a49798a9087bb969de57597dd21d6ca144fcf975b066a WatchSource:0}: Error finding container affe450ee2433e07d45a49798a9087bb969de57597dd21d6ca144fcf975b066a: Status 404 returned error can't find the container with id affe450ee2433e07d45a49798a9087bb969de57597dd21d6ca144fcf975b066a Nov 01 16:29:49.924688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.924658 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv"] Nov 01 16:29:49.933873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:49.933845 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:50.223724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.223693 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c"] Nov 01 16:29:50.224016 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:29:50.223991 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod839d2919_d17c_4e19_bdba_dcbb222ba1d1.slice/crio-0b5b6b7f689e2bc10ea996cf6452e026a5c4a53eff8892e8433cabeb01e60004 WatchSource:0}: Error finding container 0b5b6b7f689e2bc10ea996cf6452e026a5c4a53eff8892e8433cabeb01e60004: Status 404 returned error can't find the container with id 0b5b6b7f689e2bc10ea996cf6452e026a5c4a53eff8892e8433cabeb01e60004 Nov 01 16:29:50.395956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.395921 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:50.596981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.596893 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" event={"ID":"2753ad89-a39d-428b-ab90-c2923fd61ad9","Type":"ContainerStarted","Data":"8e43dc69345da5d0f6ac68d7e0cad274f3160432f3dd12f73e6c2aeedc48b651"} Nov 01 16:29:50.596981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.596944 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" event={"ID":"2753ad89-a39d-428b-ab90-c2923fd61ad9","Type":"ContainerStarted","Data":"b96d19df608ad54429b74d0a726ec0a454f973bbd19135bebcd5a385a8cd5d42"} Nov 01 16:29:50.597228 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.597011 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:29:50.598762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.598725 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" event={"ID":"839d2919-d17c-4e19-bdba-dcbb222ba1d1","Type":"ContainerStarted","Data":"4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7"} Nov 01 16:29:50.598897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.598801 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" event={"ID":"839d2919-d17c-4e19-bdba-dcbb222ba1d1","Type":"ContainerStarted","Data":"0b5b6b7f689e2bc10ea996cf6452e026a5c4a53eff8892e8433cabeb01e60004"} Nov 01 16:29:50.598897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.598822 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:29:50.600477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.600452 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" event={"ID":"7e74a675-d484-496a-a223-1d03ca77f424","Type":"ContainerStarted","Data":"9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce"} Nov 01 16:29:50.600573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.600493 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" event={"ID":"7e74a675-d484-496a-a223-1d03ca77f424","Type":"ContainerStarted","Data":"affe450ee2433e07d45a49798a9087bb969de57597dd21d6ca144fcf975b066a"} Nov 01 16:29:50.600573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.600541 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:29:50.618088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.618048 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" podStartSLOduration=1.618036909 podStartE2EDuration="1.618036909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:50.614421737 +0000 UTC m=+2528.317587675" watchObservedRunningTime="2026-11-01 16:29:50.618036909 +0000 UTC m=+2528.321202844" Nov 01 16:29:50.634123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.634084 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" podStartSLOduration=1.634073278 podStartE2EDuration="1.634073278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:50.629395853 +0000 UTC m=+2528.332561790" watchObservedRunningTime="2026-11-01 16:29:50.634073278 +0000 UTC m=+2528.337239213" Nov 01 16:29:50.648374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:50.648337 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" podStartSLOduration=1.648326882 podStartE2EDuration="1.648326882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:29:50.644536056 +0000 UTC m=+2528.347701993" watchObservedRunningTime="2026-11-01 16:29:50.648326882 +0000 UTC m=+2528.351492841" Nov 01 16:29:51.155916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:51.155884 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:51.928855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:51.928809 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:52.287559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.287469 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" podUID="5dd17200-6b81-4260-9240-9615be94a2f0" containerName="main" containerID="cri-o://c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92" gracePeriod=45 Nov 01 16:29:52.391140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.391100 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" podUID="5dd17200-6b81-4260-9240-9615be94a2f0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.236:8000/ready\": dial tcp 10.133.0.236:8000: connect: connection refused" Nov 01 16:29:52.578207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.578183 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:29:52.631884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.631850 4573 generic.go:356] "Generic (PLEG): container finished" podID="5dd17200-6b81-4260-9240-9615be94a2f0" containerID="c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92" exitCode=0 Nov 01 16:29:52.632062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.631905 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" Nov 01 16:29:52.632062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.631953 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" event={"ID":"5dd17200-6b81-4260-9240-9615be94a2f0","Type":"ContainerDied","Data":"c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92"} Nov 01 16:29:52.632062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.632008 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79" event={"ID":"5dd17200-6b81-4260-9240-9615be94a2f0","Type":"ContainerDied","Data":"d748d8a7e6d4b3ee52ec94943c7452cc8f5e4053fe67066a2b4bfd2c89365e5d"} Nov 01 16:29:52.632062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.632035 4573 scope.go:122] "RemoveContainer" containerID="c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92" Nov 01 16:29:52.645492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.645409 4573 scope.go:122] "RemoveContainer" containerID="c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92" Nov 01 16:29:52.645951 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:52.645918 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92\": container with ID starting with c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92 not found: ID does not exist" containerID="c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92" Nov 01 16:29:52.646073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.645957 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92"} err="failed to get container status \"c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92\": rpc error: code = NotFound desc = could not find container \"c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92\": container with ID starting with c1cdb91ce3444483ba369c81c34977d9d2080436138f6fe9daf81ba10f4dbe92 not found: ID does not exist" Nov 01 16:29:52.650040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.650018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:52.733081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.720265 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-dshm\") pod \"5dd17200-6b81-4260-9240-9615be94a2f0\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " Nov 01 16:29:52.733081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.720351 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-model-cache\") pod \"5dd17200-6b81-4260-9240-9615be94a2f0\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " Nov 01 16:29:52.733081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.720410 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-home\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-home\") pod \"5dd17200-6b81-4260-9240-9615be94a2f0\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " Nov 01 16:29:52.733081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.720453 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-tmp-dir\") pod \"5dd17200-6b81-4260-9240-9615be94a2f0\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " Nov 01 16:29:52.733081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.720668 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd17200-6b81-4260-9240-9615be94a2f0-kube-api-access-djpxc\" (UniqueName: \"kubernetes.io/projected/5dd17200-6b81-4260-9240-9615be94a2f0-kube-api-access-djpxc\") pod \"5dd17200-6b81-4260-9240-9615be94a2f0\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " Nov 01 16:29:52.733081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.720725 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5dd17200-6b81-4260-9240-9615be94a2f0-tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd17200-6b81-4260-9240-9615be94a2f0-tls-certs\") pod \"5dd17200-6b81-4260-9240-9615be94a2f0\" (UID: \"5dd17200-6b81-4260-9240-9615be94a2f0\") " Nov 01 16:29:52.734649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.734442 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-home" pod "5dd17200-6b81-4260-9240-9615be94a2f0" (UID: "5dd17200-6b81-4260-9240-9615be94a2f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:52.735489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.734871 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-model-cache" pod "5dd17200-6b81-4260-9240-9615be94a2f0" (UID: "5dd17200-6b81-4260-9240-9615be94a2f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:52.736131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.735805 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-tmp-dir" pod "5dd17200-6b81-4260-9240-9615be94a2f0" (UID: "5dd17200-6b81-4260-9240-9615be94a2f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:52.739830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.739790 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd17200-6b81-4260-9240-9615be94a2f0-tls-certs" pod "5dd17200-6b81-4260-9240-9615be94a2f0" (UID: "5dd17200-6b81-4260-9240-9615be94a2f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:29:52.739952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.739930 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-dshm" pod "5dd17200-6b81-4260-9240-9615be94a2f0" (UID: "5dd17200-6b81-4260-9240-9615be94a2f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:29:52.740261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.740202 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd17200-6b81-4260-9240-9615be94a2f0-kube-api-access-djpxc" pod "5dd17200-6b81-4260-9240-9615be94a2f0" (UID: "5dd17200-6b81-4260-9240-9615be94a2f0"). InnerVolumeSpecName "kube-api-access-djpxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:29:52.823500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.823426 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:52.823500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.823454 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:52.823500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.823463 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:52.823500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.823473 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djpxc\" (UniqueName: \"kubernetes.io/projected/5dd17200-6b81-4260-9240-9615be94a2f0-kube-api-access-djpxc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:52.823500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.823482 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dd17200-6b81-4260-9240-9615be94a2f0-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:52.823500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.823490 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dd17200-6b81-4260-9240-9615be94a2f0-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:29:52.957884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.957854 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79"] Nov 01 16:29:52.961725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:52.961701 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-5ccd7b/auth-test-model-5ccd7b-kserve-5c58cb5dd5-gcr79"] Nov 01 16:29:53.356539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:53.356478 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:54.069643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:54.069608 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:54.365549 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:29:54.365513 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:29:54.409696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:54.409668 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:29:54.480784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:54.479824 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd17200-6b81-4260-9240-9615be94a2f0" path="/var/lib/kubelet/pods/5dd17200-6b81-4260-9240-9615be94a2f0/volumes" Nov 01 16:29:54.866393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:54.866363 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:55.656215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:55.656166 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:56.588439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:56.588402 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:57.385339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:57.385290 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:57.579662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:57.579623 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:57.681623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:57.681531 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:57.899312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:57.899275 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:58.218596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:58.218562 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:58.867885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:58.867854 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:29:59.684701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:29:59.684671 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:00.147152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.147114 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44"] Nov 01 16:30:00.147841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.147826 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5dd17200-6b81-4260-9240-9615be94a2f0" containerName="main" Nov 01 16:30:00.147905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.147842 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd17200-6b81-4260-9240-9615be94a2f0" containerName="main" Nov 01 16:30:00.147990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.147980 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd17200-6b81-4260-9240-9615be94a2f0" containerName="main" Nov 01 16:30:00.154174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.154145 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94"] Nov 01 16:30:00.154370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.154308 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:00.160017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.159989 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:00.181945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.175938 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44"] Nov 01 16:30:00.181945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.181522 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94"] Nov 01 16:30:00.188111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.187324 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w"] Nov 01 16:30:00.215011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.214949 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p"] Nov 01 16:30:00.215412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.215383 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:00.230172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.224093 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb"] Nov 01 16:30:00.230503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.230283 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:00.242781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.241121 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w"] Nov 01 16:30:00.242781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.241189 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb"] Nov 01 16:30:00.242781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.241214 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p"] Nov 01 16:30:00.242781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.241261 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:00.245259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.245228 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b"] Nov 01 16:30:00.251479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.251449 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:00.251621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.251467 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg"] Nov 01 16:30:00.255104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.255026 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:30:00.271355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.271321 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:00.278867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.278832 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp8z\" (UniqueName: \"kubernetes.io/projected/1c48c30a-0215-4276-95a8-f1ebced71602-kube-api-access-whp8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg\" (UID: \"1c48c30a-0215-4276-95a8-f1ebced71602\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:00.279057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.278892 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-kube-api-access-6jzbq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w\" (UID: \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:00.279057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.278973 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a40d1-ad36-416e-94d5-3ca525262685-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b\" (UID: \"bb0a40d1-ad36-416e-94d5-3ca525262685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:00.279057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279018 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94\" (UID: \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:00.279251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279063 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-272bz\" (UniqueName: \"kubernetes.io/projected/7812173e-6df9-4f18-a704-b316ca57c568-kube-api-access-272bz\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44\" (UID: \"7812173e-6df9-4f18-a704-b316ca57c568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:00.279251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279125 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c48c30a-0215-4276-95a8-f1ebced71602-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg\" (UID: \"1c48c30a-0215-4276-95a8-f1ebced71602\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:00.279251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279172 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-478vv\" (UniqueName: \"kubernetes.io/projected/693232a8-19dd-44b9-ab5e-b073b3dbd82c-kube-api-access-478vv\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb\" (UID: \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:00.279251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279238 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djxwn\" (UniqueName: \"kubernetes.io/projected/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-kube-api-access-djxwn\") pod \"maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94\" (UID: \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:00.279496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279282 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/693232a8-19dd-44b9-ab5e-b073b3dbd82c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb\" (UID: \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:00.279496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279327 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812173e-6df9-4f18-a704-b316ca57c568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44\" (UID: \"7812173e-6df9-4f18-a704-b316ca57c568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:00.279496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279369 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f3372f-7731-4b2d-842c-3be1a6c71855-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p\" (UID: \"93f3372f-7731-4b2d-842c-3be1a6c71855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:00.279496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279417 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnd48\" (UniqueName: \"kubernetes.io/projected/93f3372f-7731-4b2d-842c-3be1a6c71855-kube-api-access-wnd48\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p\" (UID: \"93f3372f-7731-4b2d-842c-3be1a6c71855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:00.279496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279450 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b268r\" (UniqueName: \"kubernetes.io/projected/bb0a40d1-ad36-416e-94d5-3ca525262685-kube-api-access-b268r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b\" (UID: \"bb0a40d1-ad36-416e-94d5-3ca525262685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:00.279805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.279515 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w\" (UID: \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:00.289841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.289735 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b"] Nov 01 16:30:00.303640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.303618 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58"] Nov 01 16:30:00.310023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.310000 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:00.312504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.312479 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg"] Nov 01 16:30:00.325257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.325230 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58"] Nov 01 16:30:00.380368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380332 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp8z\" (UniqueName: \"kubernetes.io/projected/1c48c30a-0215-4276-95a8-f1ebced71602-kube-api-access-whp8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg\" (UID: \"1c48c30a-0215-4276-95a8-f1ebced71602\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:00.380368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380369 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-kube-api-access-6jzbq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w\" (UID: \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:00.380637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380487 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1343be-17df-4d00-a24b-37426a338c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58\" (UID: \"3d1343be-17df-4d00-a24b-37426a338c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:00.380637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380549 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a40d1-ad36-416e-94d5-3ca525262685-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b\" (UID: \"bb0a40d1-ad36-416e-94d5-3ca525262685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:00.380637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380620 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vndv\" (UniqueName: \"kubernetes.io/projected/3d1343be-17df-4d00-a24b-37426a338c89-kube-api-access-4vndv\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58\" (UID: \"3d1343be-17df-4d00-a24b-37426a338c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:00.380843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380659 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94\" (UID: \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:00.380843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380746 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-272bz\" (UniqueName: \"kubernetes.io/projected/7812173e-6df9-4f18-a704-b316ca57c568-kube-api-access-272bz\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44\" (UID: \"7812173e-6df9-4f18-a704-b316ca57c568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:00.380959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380847 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c48c30a-0215-4276-95a8-f1ebced71602-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg\" (UID: \"1c48c30a-0215-4276-95a8-f1ebced71602\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:00.380959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380888 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-478vv\" (UniqueName: \"kubernetes.io/projected/693232a8-19dd-44b9-ab5e-b073b3dbd82c-kube-api-access-478vv\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb\" (UID: \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:00.380959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380951 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djxwn\" (UniqueName: \"kubernetes.io/projected/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-kube-api-access-djxwn\") pod \"maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94\" (UID: \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:00.381121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.380988 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/693232a8-19dd-44b9-ab5e-b073b3dbd82c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb\" (UID: \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:00.381121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381039 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812173e-6df9-4f18-a704-b316ca57c568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44\" (UID: \"7812173e-6df9-4f18-a704-b316ca57c568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:00.381121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381077 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f3372f-7731-4b2d-842c-3be1a6c71855-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p\" (UID: \"93f3372f-7731-4b2d-842c-3be1a6c71855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:00.381277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381117 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnd48\" (UniqueName: \"kubernetes.io/projected/93f3372f-7731-4b2d-842c-3be1a6c71855-kube-api-access-wnd48\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p\" (UID: \"93f3372f-7731-4b2d-842c-3be1a6c71855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:00.381277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381144 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b268r\" (UniqueName: \"kubernetes.io/projected/bb0a40d1-ad36-416e-94d5-3ca525262685-kube-api-access-b268r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b\" (UID: \"bb0a40d1-ad36-416e-94d5-3ca525262685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:00.381277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381209 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w\" (UID: \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:00.381490 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381465 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a40d1-ad36-416e-94d5-3ca525262685-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b\" (UID: \"bb0a40d1-ad36-416e-94d5-3ca525262685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:00.381725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381704 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c48c30a-0215-4276-95a8-f1ebced71602-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg\" (UID: \"1c48c30a-0215-4276-95a8-f1ebced71602\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:00.381848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381815 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/693232a8-19dd-44b9-ab5e-b073b3dbd82c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb\" (UID: \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:00.381918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381891 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812173e-6df9-4f18-a704-b316ca57c568-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44\" (UID: \"7812173e-6df9-4f18-a704-b316ca57c568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:00.381918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381905 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f3372f-7731-4b2d-842c-3be1a6c71855-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p\" (UID: \"93f3372f-7731-4b2d-842c-3be1a6c71855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:00.382070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.381949 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94\" (UID: \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:00.382126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.382071 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w\" (UID: \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:00.390064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.390038 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-kube-api-access-6jzbq\") pod \"maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w\" (UID: \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:00.390247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.390225 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-478vv\" (UniqueName: \"kubernetes.io/projected/693232a8-19dd-44b9-ab5e-b073b3dbd82c-kube-api-access-478vv\") pod \"maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb\" (UID: \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:00.390331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.390314 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp8z\" (UniqueName: \"kubernetes.io/projected/1c48c30a-0215-4276-95a8-f1ebced71602-kube-api-access-whp8z\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg\" (UID: \"1c48c30a-0215-4276-95a8-f1ebced71602\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:00.390496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.390473 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnd48\" (UniqueName: \"kubernetes.io/projected/93f3372f-7731-4b2d-842c-3be1a6c71855-kube-api-access-wnd48\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p\" (UID: \"93f3372f-7731-4b2d-842c-3be1a6c71855\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:00.390891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.390871 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djxwn\" (UniqueName: \"kubernetes.io/projected/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-kube-api-access-djxwn\") pod \"maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94\" (UID: \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:00.391077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.391061 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-272bz\" (UniqueName: \"kubernetes.io/projected/7812173e-6df9-4f18-a704-b316ca57c568-kube-api-access-272bz\") pod \"maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44\" (UID: \"7812173e-6df9-4f18-a704-b316ca57c568\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:00.391272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.391250 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b268r\" (UniqueName: \"kubernetes.io/projected/bb0a40d1-ad36-416e-94d5-3ca525262685-kube-api-access-b268r\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b\" (UID: \"bb0a40d1-ad36-416e-94d5-3ca525262685\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:00.482666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.482583 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1343be-17df-4d00-a24b-37426a338c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58\" (UID: \"3d1343be-17df-4d00-a24b-37426a338c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:00.482666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.482641 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vndv\" (UniqueName: \"kubernetes.io/projected/3d1343be-17df-4d00-a24b-37426a338c89-kube-api-access-4vndv\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58\" (UID: \"3d1343be-17df-4d00-a24b-37426a338c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:00.483217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.483196 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1343be-17df-4d00-a24b-37426a338c89-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58\" (UID: \"3d1343be-17df-4d00-a24b-37426a338c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:00.488040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.488021 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:00.490825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.490792 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vndv\" (UniqueName: \"kubernetes.io/projected/3d1343be-17df-4d00-a24b-37426a338c89-kube-api-access-4vndv\") pod \"maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58\" (UID: \"3d1343be-17df-4d00-a24b-37426a338c89\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:00.508690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.508661 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:00.543903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.543878 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:00.584807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.584700 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:00.592673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.590963 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:00.608119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.607926 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:00.624605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.624382 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:00.635396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.635357 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:00.644579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.644343 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:00.785905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:00.785872 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44"] Nov 01 16:30:00.787071 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:00.787044 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7812173e_6df9_4f18_a704_b316ca57c568.slice/crio-6b520c76be291fdd17e9cf4ad3ef2b21d69e48882313a484ba9deeeff6d6c1cb WatchSource:0}: Error finding container 6b520c76be291fdd17e9cf4ad3ef2b21d69e48882313a484ba9deeeff6d6c1cb: Status 404 returned error can't find the container with id 6b520c76be291fdd17e9cf4ad3ef2b21d69e48882313a484ba9deeeff6d6c1cb Nov 01 16:30:01.022718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.022663 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94"] Nov 01 16:30:01.024060 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:01.024027 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eaa91b1_d9bf_4ab2_8b66_1a3588c94ee6.slice/crio-677055584bf0c4ac1ce42f9f9a4dc5f95a421a2a193cba4b6334f0dddb8a3116 WatchSource:0}: Error finding container 677055584bf0c4ac1ce42f9f9a4dc5f95a421a2a193cba4b6334f0dddb8a3116: Status 404 returned error can't find the container with id 677055584bf0c4ac1ce42f9f9a4dc5f95a421a2a193cba4b6334f0dddb8a3116 Nov 01 16:30:01.024982 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:01.024957 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693232a8_19dd_44b9_ab5e_b073b3dbd82c.slice/crio-df81b7a04519bd5e2d18113a453d9c2df71289647f72e649ee1d3d55d21ae605 WatchSource:0}: Error finding container df81b7a04519bd5e2d18113a453d9c2df71289647f72e649ee1d3d55d21ae605: Status 404 returned error can't find the container with id df81b7a04519bd5e2d18113a453d9c2df71289647f72e649ee1d3d55d21ae605 Nov 01 16:30:01.028944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.028921 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb"] Nov 01 16:30:01.095508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.095477 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg"] Nov 01 16:30:01.098997 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:01.098933 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c48c30a_0215_4276_95a8_f1ebced71602.slice/crio-89b65450653a027957c3845530e27afa056e4853f4f0ab83d75a64a8d8fa2466 WatchSource:0}: Error finding container 89b65450653a027957c3845530e27afa056e4853f4f0ab83d75a64a8d8fa2466: Status 404 returned error can't find the container with id 89b65450653a027957c3845530e27afa056e4853f4f0ab83d75a64a8d8fa2466 Nov 01 16:30:01.102699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.101784 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b"] Nov 01 16:30:01.105638 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:01.105613 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0a40d1_ad36_416e_94d5_3ca525262685.slice/crio-444c72bdfd8ab585e681d6a7454b33b72e56a9c3149551a4efb2a6aab3ccb807 WatchSource:0}: Error finding container 444c72bdfd8ab585e681d6a7454b33b72e56a9c3149551a4efb2a6aab3ccb807: Status 404 returned error can't find the container with id 444c72bdfd8ab585e681d6a7454b33b72e56a9c3149551a4efb2a6aab3ccb807 Nov 01 16:30:01.113568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.113545 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58"] Nov 01 16:30:01.114074 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:01.114048 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ae3328_ea4c_43ff_87a7_2a6b6e9a347d.slice/crio-786bd3a2d5d0c092a76f7118009f36267055b7ea2fa46384f4e47076ab2eebfc WatchSource:0}: Error finding container 786bd3a2d5d0c092a76f7118009f36267055b7ea2fa46384f4e47076ab2eebfc: Status 404 returned error can't find the container with id 786bd3a2d5d0c092a76f7118009f36267055b7ea2fa46384f4e47076ab2eebfc Nov 01 16:30:01.121052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.121023 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w"] Nov 01 16:30:01.122215 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:01.122186 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1343be_17df_4d00_a24b_37426a338c89.slice/crio-7a5c22af2eafd166485de46a5a15548127dc7b44e1a94877303dce54d1048622 WatchSource:0}: Error finding container 7a5c22af2eafd166485de46a5a15548127dc7b44e1a94877303dce54d1048622: Status 404 returned error can't find the container with id 7a5c22af2eafd166485de46a5a15548127dc7b44e1a94877303dce54d1048622 Nov 01 16:30:01.173465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.173437 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p"] Nov 01 16:30:01.367941 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:01.367897 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f3372f_7731_4b2d_842c_3be1a6c71855.slice/crio-4557c2e1398148625c451ede341b79316c9e5e4fe1d183ee866be74c05f8fbf5 WatchSource:0}: Error finding container 4557c2e1398148625c451ede341b79316c9e5e4fe1d183ee866be74c05f8fbf5: Status 404 returned error can't find the container with id 4557c2e1398148625c451ede341b79316c9e5e4fe1d183ee866be74c05f8fbf5 Nov 01 16:30:01.464069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.464030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:01.610722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.610681 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:30:01.611666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.611637 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:30:01.611862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.611720 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:30:01.755067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.755025 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" event={"ID":"93f3372f-7731-4b2d-842c-3be1a6c71855","Type":"ContainerStarted","Data":"4557c2e1398148625c451ede341b79316c9e5e4fe1d183ee866be74c05f8fbf5"} Nov 01 16:30:01.757021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.756989 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" event={"ID":"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d","Type":"ContainerStarted","Data":"786bd3a2d5d0c092a76f7118009f36267055b7ea2fa46384f4e47076ab2eebfc"} Nov 01 16:30:01.769828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.769170 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" event={"ID":"693232a8-19dd-44b9-ab5e-b073b3dbd82c","Type":"ContainerStarted","Data":"18d65d550759b19b3ca12f25d4bf12609d31d30581c87b29f03d704e91a52ad9"} Nov 01 16:30:01.769828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.769229 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" event={"ID":"693232a8-19dd-44b9-ab5e-b073b3dbd82c","Type":"ContainerStarted","Data":"df81b7a04519bd5e2d18113a453d9c2df71289647f72e649ee1d3d55d21ae605"} Nov 01 16:30:01.773194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.772856 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" event={"ID":"7812173e-6df9-4f18-a704-b316ca57c568","Type":"ContainerStarted","Data":"25a75bb37b3f848ef1a214bda62cc92544f950d9f332d9fa8911a5f8ec8203bc"} Nov 01 16:30:01.773194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.772907 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" event={"ID":"7812173e-6df9-4f18-a704-b316ca57c568","Type":"ContainerStarted","Data":"6b520c76be291fdd17e9cf4ad3ef2b21d69e48882313a484ba9deeeff6d6c1cb"} Nov 01 16:30:01.774957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.774928 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" event={"ID":"bb0a40d1-ad36-416e-94d5-3ca525262685","Type":"ContainerStarted","Data":"444c72bdfd8ab585e681d6a7454b33b72e56a9c3149551a4efb2a6aab3ccb807"} Nov 01 16:30:01.776864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.776836 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" event={"ID":"1c48c30a-0215-4276-95a8-f1ebced71602","Type":"ContainerStarted","Data":"b18cd59d5e4a4bb5396c53bb915913b7e1cbd71a553ec89a728ee92b8a7b1c93"} Nov 01 16:30:01.776988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.776886 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" event={"ID":"1c48c30a-0215-4276-95a8-f1ebced71602","Type":"ContainerStarted","Data":"89b65450653a027957c3845530e27afa056e4853f4f0ab83d75a64a8d8fa2466"} Nov 01 16:30:01.778642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.778614 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" event={"ID":"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6","Type":"ContainerStarted","Data":"960a5b21818daadf189753fa317ea508e753110dfe39e4b6d0ffb1a034eebbac"} Nov 01 16:30:01.778724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.778670 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" event={"ID":"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6","Type":"ContainerStarted","Data":"677055584bf0c4ac1ce42f9f9a4dc5f95a421a2a193cba4b6334f0dddb8a3116"} Nov 01 16:30:01.781075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.781051 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" event={"ID":"3d1343be-17df-4d00-a24b-37426a338c89","Type":"ContainerStarted","Data":"7a5c22af2eafd166485de46a5a15548127dc7b44e1a94877303dce54d1048622"} Nov 01 16:30:01.789826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.789770 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" podStartSLOduration=1.789737787 podStartE2EDuration="1.789737787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:01.786027912 +0000 UTC m=+2539.489193848" watchObservedRunningTime="2026-11-01 16:30:01.789737787 +0000 UTC m=+2539.492903722" Nov 01 16:30:01.803427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.803372 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" podStartSLOduration=1.8033583069999999 podStartE2EDuration="1.803358307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:01.798939627 +0000 UTC m=+2539.502105564" watchObservedRunningTime="2026-11-01 16:30:01.803358307 +0000 UTC m=+2539.506524242" Nov 01 16:30:01.815207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.815153 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" podStartSLOduration=1.815140177 podStartE2EDuration="1.815140177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:01.812710962 +0000 UTC m=+2539.515876900" watchObservedRunningTime="2026-11-01 16:30:01.815140177 +0000 UTC m=+2539.518306112" Nov 01 16:30:01.844241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:01.844105 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" podStartSLOduration=1.844085299 podStartE2EDuration="1.844085299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:01.830547675 +0000 UTC m=+2539.533713612" watchObservedRunningTime="2026-11-01 16:30:01.844085299 +0000 UTC m=+2539.547251236" Nov 01 16:30:02.418530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.418445 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:02.791173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.791138 4573 generic.go:356] "Generic (PLEG): container finished" podID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerID="874c505d0d9afead2aad89a443ead359c20214b52a50e1702e6dc4aa8b9d785b" exitCode=60 Nov 01 16:30:02.791373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.791230 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" event={"ID":"bb0a40d1-ad36-416e-94d5-3ca525262685","Type":"ContainerDied","Data":"874c505d0d9afead2aad89a443ead359c20214b52a50e1702e6dc4aa8b9d785b"} Nov 01 16:30:02.791574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.791554 4573 scope.go:122] "RemoveContainer" containerID="874c505d0d9afead2aad89a443ead359c20214b52a50e1702e6dc4aa8b9d785b" Nov 01 16:30:02.794032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.794006 4573 generic.go:356] "Generic (PLEG): container finished" podID="1c48c30a-0215-4276-95a8-f1ebced71602" containerID="b18cd59d5e4a4bb5396c53bb915913b7e1cbd71a553ec89a728ee92b8a7b1c93" exitCode=60 Nov 01 16:30:02.794177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.794078 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" event={"ID":"1c48c30a-0215-4276-95a8-f1ebced71602","Type":"ContainerDied","Data":"b18cd59d5e4a4bb5396c53bb915913b7e1cbd71a553ec89a728ee92b8a7b1c93"} Nov 01 16:30:02.794420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.794399 4573 scope.go:122] "RemoveContainer" containerID="b18cd59d5e4a4bb5396c53bb915913b7e1cbd71a553ec89a728ee92b8a7b1c93" Nov 01 16:30:02.796053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.796028 4573 generic.go:356] "Generic (PLEG): container finished" podID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerID="960a5b21818daadf189753fa317ea508e753110dfe39e4b6d0ffb1a034eebbac" exitCode=60 Nov 01 16:30:02.796383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.796365 4573 scope.go:122] "RemoveContainer" containerID="960a5b21818daadf189753fa317ea508e753110dfe39e4b6d0ffb1a034eebbac" Nov 01 16:30:02.796955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.796934 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" event={"ID":"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6","Type":"ContainerDied","Data":"960a5b21818daadf189753fa317ea508e753110dfe39e4b6d0ffb1a034eebbac"} Nov 01 16:30:02.798072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.798050 4573 generic.go:356] "Generic (PLEG): container finished" podID="3d1343be-17df-4d00-a24b-37426a338c89" containerID="dd9cac37abd327ae17678db925f8c56f40e62951992e1ca437f9e6076769b53e" exitCode=60 Nov 01 16:30:02.798180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.798072 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" event={"ID":"3d1343be-17df-4d00-a24b-37426a338c89","Type":"ContainerDied","Data":"dd9cac37abd327ae17678db925f8c56f40e62951992e1ca437f9e6076769b53e"} Nov 01 16:30:02.798428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.798399 4573 scope.go:122] "RemoveContainer" containerID="dd9cac37abd327ae17678db925f8c56f40e62951992e1ca437f9e6076769b53e" Nov 01 16:30:02.799909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.799890 4573 generic.go:356] "Generic (PLEG): container finished" podID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerID="8413676e2ba7c58a1095fd6fd3b84dff2691c4633b405c780b822e62e9829168" exitCode=60 Nov 01 16:30:02.800015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.799947 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" event={"ID":"93f3372f-7731-4b2d-842c-3be1a6c71855","Type":"ContainerDied","Data":"8413676e2ba7c58a1095fd6fd3b84dff2691c4633b405c780b822e62e9829168"} Nov 01 16:30:02.800249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.800233 4573 scope.go:122] "RemoveContainer" containerID="8413676e2ba7c58a1095fd6fd3b84dff2691c4633b405c780b822e62e9829168" Nov 01 16:30:02.802035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.802014 4573 generic.go:356] "Generic (PLEG): container finished" podID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerID="fdb285b01b7b01561fe4058acf34d16a26daabf6438517b2dce1ec851ad5fdb0" exitCode=60 Nov 01 16:30:02.802126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.802090 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" event={"ID":"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d","Type":"ContainerDied","Data":"fdb285b01b7b01561fe4058acf34d16a26daabf6438517b2dce1ec851ad5fdb0"} Nov 01 16:30:02.802293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.802279 4573 scope.go:122] "RemoveContainer" containerID="fdb285b01b7b01561fe4058acf34d16a26daabf6438517b2dce1ec851ad5fdb0" Nov 01 16:30:02.809318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.804658 4573 generic.go:356] "Generic (PLEG): container finished" podID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerID="18d65d550759b19b3ca12f25d4bf12609d31d30581c87b29f03d704e91a52ad9" exitCode=60 Nov 01 16:30:02.809318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.804728 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" event={"ID":"693232a8-19dd-44b9-ab5e-b073b3dbd82c","Type":"ContainerDied","Data":"18d65d550759b19b3ca12f25d4bf12609d31d30581c87b29f03d704e91a52ad9"} Nov 01 16:30:02.809318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.805108 4573 scope.go:122] "RemoveContainer" containerID="18d65d550759b19b3ca12f25d4bf12609d31d30581c87b29f03d704e91a52ad9" Nov 01 16:30:02.817389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.817367 4573 generic.go:356] "Generic (PLEG): container finished" podID="7812173e-6df9-4f18-a704-b316ca57c568" containerID="25a75bb37b3f848ef1a214bda62cc92544f950d9f332d9fa8911a5f8ec8203bc" exitCode=60 Nov 01 16:30:02.817476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.817413 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" event={"ID":"7812173e-6df9-4f18-a704-b316ca57c568","Type":"ContainerDied","Data":"25a75bb37b3f848ef1a214bda62cc92544f950d9f332d9fa8911a5f8ec8203bc"} Nov 01 16:30:02.817844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:02.817710 4573 scope.go:122] "RemoveContainer" containerID="25a75bb37b3f848ef1a214bda62cc92544f950d9f332d9fa8911a5f8ec8203bc" Nov 01 16:30:03.154634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.154600 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:03.352358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.352332 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm_df545125-bbe8-4ed1-8d81-012278b7eeaa/payload-processing/0.log" Nov 01 16:30:03.827308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.827210 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" event={"ID":"3d1343be-17df-4d00-a24b-37426a338c89","Type":"ContainerStarted","Data":"fa522e39c4d4cc84aa7513fa9782d4481e4a440043f8279b0afade51596c244c"} Nov 01 16:30:03.829395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.829367 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" event={"ID":"93f3372f-7731-4b2d-842c-3be1a6c71855","Type":"ContainerStarted","Data":"6e127073cacb2fe9aeb021a8924cf2afa01a3bb23de63be6ec9dc0f97bb9c572"} Nov 01 16:30:03.831534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.831508 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" event={"ID":"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d","Type":"ContainerStarted","Data":"36301111b007ec03b7d6b056c95ec99eee6208ad78ce3d586c0ffc018fada3d7"} Nov 01 16:30:03.833513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.833485 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" event={"ID":"693232a8-19dd-44b9-ab5e-b073b3dbd82c","Type":"ContainerStarted","Data":"f99b9e1c21e09bed99979f8d1074404e1a4bbdbae2e9c38563b91385a80917ed"} Nov 01 16:30:03.835381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.835358 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" event={"ID":"7812173e-6df9-4f18-a704-b316ca57c568","Type":"ContainerStarted","Data":"2a920de98c9855b2facea6e58a516bb7f2f9e76d3cf47424ef1e54e11a4e3799"} Nov 01 16:30:03.837318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.837300 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" event={"ID":"bb0a40d1-ad36-416e-94d5-3ca525262685","Type":"ContainerStarted","Data":"49dec7accf92d4af2e2943a89b03e2b6390af0e0f87f5d7ba3c38a46c52b9660"} Nov 01 16:30:03.839187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.839164 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" event={"ID":"1c48c30a-0215-4276-95a8-f1ebced71602","Type":"ContainerStarted","Data":"53d84369a77ee43c49d319f7d90a1e3d3a0a64f9439dc3a63ba6065199e172a9"} Nov 01 16:30:03.841088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.841067 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" event={"ID":"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6","Type":"ContainerStarted","Data":"9c43b4e8cf8c4a2a2a1a50016048c756ac6471f2a559dee83cbf56e5093a8a31"} Nov 01 16:30:03.865421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.865395 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l_d866f151-f5b8-4f65-b376-24e6c0da561e/payload-processing/0.log" Nov 01 16:30:03.865856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.865807 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" podStartSLOduration=3.865789688 podStartE2EDuration="3.865789688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:03.84693925 +0000 UTC m=+2541.550105192" watchObservedRunningTime="2026-11-01 16:30:03.865789688 +0000 UTC m=+2541.568955626" Nov 01 16:30:03.899491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.899436 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" podStartSLOduration=3.899421909 podStartE2EDuration="3.899421909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:03.898908374 +0000 UTC m=+2541.602074309" watchObservedRunningTime="2026-11-01 16:30:03.899421909 +0000 UTC m=+2541.602587844" Nov 01 16:30:03.951008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.950976 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs"] Nov 01 16:30:03.955619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.955600 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:03.961975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.961919 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" podStartSLOduration=3.961902002 podStartE2EDuration="3.961902002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:03.958934098 +0000 UTC m=+2541.662100033" watchObservedRunningTime="2026-11-01 16:30:03.961902002 +0000 UTC m=+2541.665067951" Nov 01 16:30:03.967601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.967557 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-028c46-serving-cert\"" Nov 01 16:30:03.977893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:03.977862 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs"] Nov 01 16:30:04.009002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.008954 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" podStartSLOduration=4.008938327 podStartE2EDuration="4.008938327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:03.998667006 +0000 UTC m=+2541.701832941" watchObservedRunningTime="2026-11-01 16:30:04.008938327 +0000 UTC m=+2541.712104288" Nov 01 16:30:04.043899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.043873 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l_d866f151-f5b8-4f65-b376-24e6c0da561e/payload-processing/0.log" Nov 01 16:30:04.049615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.049582 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmbx4\" (UniqueName: \"kubernetes.io/projected/a1b57bd1-e85b-46bb-a367-d921b85a7201-kube-api-access-vmbx4\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.049744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.049712 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.049841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.049803 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b57bd1-e85b-46bb-a367-d921b85a7201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.050185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.050153 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-metrics-certs\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.122055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.122024 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:04.152164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.152131 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.152331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.152198 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b57bd1-e85b-46bb-a367-d921b85a7201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.152331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.152283 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-metrics-certs\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.152408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.152364 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmbx4\" (UniqueName: \"kubernetes.io/projected/a1b57bd1-e85b-46bb-a367-d921b85a7201-kube-api-access-vmbx4\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.152983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.152960 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b57bd1-e85b-46bb-a367-d921b85a7201-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.154653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.154624 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-maas-api-tls\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.154773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.154674 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-metrics-certs\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.207521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.207444 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmbx4\" (UniqueName: \"kubernetes.io/projected/a1b57bd1-e85b-46bb-a367-d921b85a7201-kube-api-access-vmbx4\") pod \"maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.217453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.217278 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:04.353450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.353379 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:04.710602 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:04.710563 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b57bd1_e85b_46bb_a367_d921b85a7201.slice/crio-dd01344dd4c9addc8bcb5b774554f4b143b75cfa041d11ef4c365bc852685fe1 WatchSource:0}: Error finding container dd01344dd4c9addc8bcb5b774554f4b143b75cfa041d11ef4c365bc852685fe1: Status 404 returned error can't find the container with id dd01344dd4c9addc8bcb5b774554f4b143b75cfa041d11ef4c365bc852685fe1 Nov 01 16:30:04.724550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.724515 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs"] Nov 01 16:30:04.852343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.852306 4573 generic.go:356] "Generic (PLEG): container finished" podID="3d1343be-17df-4d00-a24b-37426a338c89" containerID="fa522e39c4d4cc84aa7513fa9782d4481e4a440043f8279b0afade51596c244c" exitCode=60 Nov 01 16:30:04.852833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.852395 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" event={"ID":"3d1343be-17df-4d00-a24b-37426a338c89","Type":"ContainerDied","Data":"fa522e39c4d4cc84aa7513fa9782d4481e4a440043f8279b0afade51596c244c"} Nov 01 16:30:04.852833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.852453 4573 scope.go:122] "RemoveContainer" containerID="dd9cac37abd327ae17678db925f8c56f40e62951992e1ca437f9e6076769b53e" Nov 01 16:30:04.852833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.852711 4573 scope.go:122] "RemoveContainer" containerID="fa522e39c4d4cc84aa7513fa9782d4481e4a440043f8279b0afade51596c244c" Nov 01 16:30:04.853145 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:04.853124 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58_odh-ai-gateway-infra(3d1343be-17df-4d00-a24b-37426a338c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" podUID="3d1343be-17df-4d00-a24b-37426a338c89" Nov 01 16:30:04.854560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.854541 4573 generic.go:356] "Generic (PLEG): container finished" podID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerID="6e127073cacb2fe9aeb021a8924cf2afa01a3bb23de63be6ec9dc0f97bb9c572" exitCode=60 Nov 01 16:30:04.854651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.854609 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" event={"ID":"93f3372f-7731-4b2d-842c-3be1a6c71855","Type":"ContainerDied","Data":"6e127073cacb2fe9aeb021a8924cf2afa01a3bb23de63be6ec9dc0f97bb9c572"} Nov 01 16:30:04.854921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.854904 4573 scope.go:122] "RemoveContainer" containerID="6e127073cacb2fe9aeb021a8924cf2afa01a3bb23de63be6ec9dc0f97bb9c572" Nov 01 16:30:04.855207 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:04.855189 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p_odh-ai-gateway-infra(93f3372f-7731-4b2d-842c-3be1a6c71855)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" Nov 01 16:30:04.856704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.856682 4573 generic.go:356] "Generic (PLEG): container finished" podID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerID="36301111b007ec03b7d6b056c95ec99eee6208ad78ce3d586c0ffc018fada3d7" exitCode=60 Nov 01 16:30:04.856839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.856778 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" event={"ID":"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d","Type":"ContainerDied","Data":"36301111b007ec03b7d6b056c95ec99eee6208ad78ce3d586c0ffc018fada3d7"} Nov 01 16:30:04.857199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.857184 4573 scope.go:122] "RemoveContainer" containerID="36301111b007ec03b7d6b056c95ec99eee6208ad78ce3d586c0ffc018fada3d7" Nov 01 16:30:04.857455 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:04.857433 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w_odh-ai-gateway-infra(e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" Nov 01 16:30:04.858940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.858922 4573 generic.go:356] "Generic (PLEG): container finished" podID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerID="f99b9e1c21e09bed99979f8d1074404e1a4bbdbae2e9c38563b91385a80917ed" exitCode=60 Nov 01 16:30:04.859041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.858981 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" event={"ID":"693232a8-19dd-44b9-ab5e-b073b3dbd82c","Type":"ContainerDied","Data":"f99b9e1c21e09bed99979f8d1074404e1a4bbdbae2e9c38563b91385a80917ed"} Nov 01 16:30:04.859230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.859212 4573 scope.go:122] "RemoveContainer" containerID="f99b9e1c21e09bed99979f8d1074404e1a4bbdbae2e9c38563b91385a80917ed" Nov 01 16:30:04.859492 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:04.859469 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb_odh-ai-gateway-infra(693232a8-19dd-44b9-ab5e-b073b3dbd82c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" Nov 01 16:30:04.861007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.860989 4573 generic.go:356] "Generic (PLEG): container finished" podID="7812173e-6df9-4f18-a704-b316ca57c568" containerID="2a920de98c9855b2facea6e58a516bb7f2f9e76d3cf47424ef1e54e11a4e3799" exitCode=60 Nov 01 16:30:04.861149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.861069 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" event={"ID":"7812173e-6df9-4f18-a704-b316ca57c568","Type":"ContainerDied","Data":"2a920de98c9855b2facea6e58a516bb7f2f9e76d3cf47424ef1e54e11a4e3799"} Nov 01 16:30:04.861299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.861279 4573 scope.go:122] "RemoveContainer" containerID="2a920de98c9855b2facea6e58a516bb7f2f9e76d3cf47424ef1e54e11a4e3799" Nov 01 16:30:04.861543 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:04.861519 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44_odh-ai-gateway-infra(7812173e-6df9-4f18-a704-b316ca57c568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" podUID="7812173e-6df9-4f18-a704-b316ca57c568" Nov 01 16:30:04.863013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.862900 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" event={"ID":"a1b57bd1-e85b-46bb-a367-d921b85a7201","Type":"ContainerStarted","Data":"dd01344dd4c9addc8bcb5b774554f4b143b75cfa041d11ef4c365bc852685fe1"} Nov 01 16:30:04.895031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.894998 4573 generic.go:356] "Generic (PLEG): container finished" podID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerID="49dec7accf92d4af2e2943a89b03e2b6390af0e0f87f5d7ba3c38a46c52b9660" exitCode=60 Nov 01 16:30:04.895236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.895108 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" event={"ID":"bb0a40d1-ad36-416e-94d5-3ca525262685","Type":"ContainerDied","Data":"49dec7accf92d4af2e2943a89b03e2b6390af0e0f87f5d7ba3c38a46c52b9660"} Nov 01 16:30:04.895439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.895419 4573 scope.go:122] "RemoveContainer" containerID="49dec7accf92d4af2e2943a89b03e2b6390af0e0f87f5d7ba3c38a46c52b9660" Nov 01 16:30:04.895890 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:04.895860 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b_odh-ai-gateway-infra(bb0a40d1-ad36-416e-94d5-3ca525262685)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" Nov 01 16:30:04.896866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.896848 4573 scope.go:122] "RemoveContainer" containerID="8413676e2ba7c58a1095fd6fd3b84dff2691c4633b405c780b822e62e9829168" Nov 01 16:30:04.897544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.897519 4573 generic.go:356] "Generic (PLEG): container finished" podID="1c48c30a-0215-4276-95a8-f1ebced71602" containerID="53d84369a77ee43c49d319f7d90a1e3d3a0a64f9439dc3a63ba6065199e172a9" exitCode=60 Nov 01 16:30:04.897668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.897595 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" event={"ID":"1c48c30a-0215-4276-95a8-f1ebced71602","Type":"ContainerDied","Data":"53d84369a77ee43c49d319f7d90a1e3d3a0a64f9439dc3a63ba6065199e172a9"} Nov 01 16:30:04.897967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.897941 4573 scope.go:122] "RemoveContainer" containerID="53d84369a77ee43c49d319f7d90a1e3d3a0a64f9439dc3a63ba6065199e172a9" Nov 01 16:30:04.898217 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:04.898186 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg_odh-ai-gateway-infra(1c48c30a-0215-4276-95a8-f1ebced71602)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" Nov 01 16:30:04.900296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.900275 4573 generic.go:356] "Generic (PLEG): container finished" podID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerID="9c43b4e8cf8c4a2a2a1a50016048c756ac6471f2a559dee83cbf56e5093a8a31" exitCode=60 Nov 01 16:30:04.900410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.900372 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" event={"ID":"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6","Type":"ContainerDied","Data":"9c43b4e8cf8c4a2a2a1a50016048c756ac6471f2a559dee83cbf56e5093a8a31"} Nov 01 16:30:04.900651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.900636 4573 scope.go:122] "RemoveContainer" containerID="9c43b4e8cf8c4a2a2a1a50016048c756ac6471f2a559dee83cbf56e5093a8a31" Nov 01 16:30:04.900930 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:04.900905 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94_odh-ai-gateway-infra(3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" Nov 01 16:30:04.916056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.916029 4573 scope.go:122] "RemoveContainer" containerID="fdb285b01b7b01561fe4058acf34d16a26daabf6438517b2dce1ec851ad5fdb0" Nov 01 16:30:04.932350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.932329 4573 scope.go:122] "RemoveContainer" containerID="18d65d550759b19b3ca12f25d4bf12609d31d30581c87b29f03d704e91a52ad9" Nov 01 16:30:04.947174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.947146 4573 scope.go:122] "RemoveContainer" containerID="25a75bb37b3f848ef1a214bda62cc92544f950d9f332d9fa8911a5f8ec8203bc" Nov 01 16:30:04.960918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.960894 4573 scope.go:122] "RemoveContainer" containerID="874c505d0d9afead2aad89a443ead359c20214b52a50e1702e6dc4aa8b9d785b" Nov 01 16:30:04.975680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.975660 4573 scope.go:122] "RemoveContainer" containerID="b18cd59d5e4a4bb5396c53bb915913b7e1cbd71a553ec89a728ee92b8a7b1c93" Nov 01 16:30:04.989621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:04.989597 4573 scope.go:122] "RemoveContainer" containerID="960a5b21818daadf189753fa317ea508e753110dfe39e4b6d0ffb1a034eebbac" Nov 01 16:30:05.047207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.047174 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:05.238990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.238965 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:05.444404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.444361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:05.681367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.681328 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:05.895657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.895626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:05.910916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.910891 4573 scope.go:122] "RemoveContainer" containerID="fa522e39c4d4cc84aa7513fa9782d4481e4a440043f8279b0afade51596c244c" Nov 01 16:30:05.911311 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:05.911271 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58_odh-ai-gateway-infra(3d1343be-17df-4d00-a24b-37426a338c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" podUID="3d1343be-17df-4d00-a24b-37426a338c89" Nov 01 16:30:05.912612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.912588 4573 scope.go:122] "RemoveContainer" containerID="6e127073cacb2fe9aeb021a8924cf2afa01a3bb23de63be6ec9dc0f97bb9c572" Nov 01 16:30:05.912955 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:05.912935 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p_odh-ai-gateway-infra(93f3372f-7731-4b2d-842c-3be1a6c71855)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" Nov 01 16:30:05.914316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.914295 4573 scope.go:122] "RemoveContainer" containerID="36301111b007ec03b7d6b056c95ec99eee6208ad78ce3d586c0ffc018fada3d7" Nov 01 16:30:05.914581 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:05.914554 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w_odh-ai-gateway-infra(e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" Nov 01 16:30:05.916005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.915988 4573 scope.go:122] "RemoveContainer" containerID="f99b9e1c21e09bed99979f8d1074404e1a4bbdbae2e9c38563b91385a80917ed" Nov 01 16:30:05.916201 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:05.916181 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb_odh-ai-gateway-infra(693232a8-19dd-44b9-ab5e-b073b3dbd82c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" Nov 01 16:30:05.917614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.917595 4573 scope.go:122] "RemoveContainer" containerID="2a920de98c9855b2facea6e58a516bb7f2f9e76d3cf47424ef1e54e11a4e3799" Nov 01 16:30:05.917874 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:05.917851 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44_odh-ai-gateway-infra(7812173e-6df9-4f18-a704-b316ca57c568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" podUID="7812173e-6df9-4f18-a704-b316ca57c568" Nov 01 16:30:05.918953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.918933 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" event={"ID":"a1b57bd1-e85b-46bb-a367-d921b85a7201","Type":"ContainerStarted","Data":"5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899"} Nov 01 16:30:05.919042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.919025 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:05.920970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.920954 4573 scope.go:122] "RemoveContainer" containerID="49dec7accf92d4af2e2943a89b03e2b6390af0e0f87f5d7ba3c38a46c52b9660" Nov 01 16:30:05.921175 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:05.921157 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b_odh-ai-gateway-infra(bb0a40d1-ad36-416e-94d5-3ca525262685)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" Nov 01 16:30:05.922658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.922641 4573 scope.go:122] "RemoveContainer" containerID="53d84369a77ee43c49d319f7d90a1e3d3a0a64f9439dc3a63ba6065199e172a9" Nov 01 16:30:05.922902 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:05.922880 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg_odh-ai-gateway-infra(1c48c30a-0215-4276-95a8-f1ebced71602)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" Nov 01 16:30:05.924459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.924443 4573 scope.go:122] "RemoveContainer" containerID="9c43b4e8cf8c4a2a2a1a50016048c756ac6471f2a559dee83cbf56e5093a8a31" Nov 01 16:30:05.924646 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:05.924628 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94_odh-ai-gateway-infra(3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" Nov 01 16:30:05.999328 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:05.999234 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:06.045227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:06.045169 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" podStartSLOduration=2.557556053 podStartE2EDuration="3.045152501s" totalImagesPullingTime="487.596448ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:06.043019823 +0000 UTC m=+2543.746185759" watchObservedRunningTime="2026-11-01 16:30:06.045152501 +0000 UTC m=+2543.748318441" Nov 01 16:30:06.228835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:06.228229 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:06.506370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:06.506337 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:07.660692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:07.660659 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:08.328404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:08.328347 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:09.226111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:09.226073 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:09.883597 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:09.883567 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:09.974664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:09.974631 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:10.755528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:10.755498 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:11.105955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.105863 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:11.212572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.212540 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:11.220585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.220555 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x"] Nov 01 16:30:11.226161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.226132 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:11.239137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.239107 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6b6cfc-dockercfg-gdkq6\"" Nov 01 16:30:11.239322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.239204 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6b6cfc\"" Nov 01 16:30:11.248891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.248317 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x"] Nov 01 16:30:11.317822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.307525 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq8wt\" (UniqueName: \"kubernetes.io/projected/09f12a70-d45e-4340-bdaf-c61d3af3d130-kube-api-access-qq8wt\") pod \"payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x\" (UID: \"09f12a70-d45e-4340-bdaf-c61d3af3d130\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:11.317822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.307606 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f12a70-d45e-4340-bdaf-c61d3af3d130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x\" (UID: \"09f12a70-d45e-4340-bdaf-c61d3af3d130\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:11.414283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.414241 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8wt\" (UniqueName: \"kubernetes.io/projected/09f12a70-d45e-4340-bdaf-c61d3af3d130-kube-api-access-qq8wt\") pod \"payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x\" (UID: \"09f12a70-d45e-4340-bdaf-c61d3af3d130\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:11.433233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.433193 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f12a70-d45e-4340-bdaf-c61d3af3d130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x\" (UID: \"09f12a70-d45e-4340-bdaf-c61d3af3d130\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:11.434883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.434844 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f12a70-d45e-4340-bdaf-c61d3af3d130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x\" (UID: \"09f12a70-d45e-4340-bdaf-c61d3af3d130\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:11.441451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.441425 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8wt\" (UniqueName: \"kubernetes.io/projected/09f12a70-d45e-4340-bdaf-c61d3af3d130-kube-api-access-qq8wt\") pod \"payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x\" (UID: \"09f12a70-d45e-4340-bdaf-c61d3af3d130\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:11.545721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.545685 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:11.806639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.806553 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:11.870342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.865381 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x"] Nov 01 16:30:11.874647 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:11.874612 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f12a70_d45e_4340_bdaf_c61d3af3d130.slice/crio-6898ecaa60dc086209a460636cd913e63883c4c7d3f9627cc1fc4369ea087e32 WatchSource:0}: Error finding container 6898ecaa60dc086209a460636cd913e63883c4c7d3f9627cc1fc4369ea087e32: Status 404 returned error can't find the container with id 6898ecaa60dc086209a460636cd913e63883c4c7d3f9627cc1fc4369ea087e32 Nov 01 16:30:11.938542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.938521 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:30:11.976995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.976955 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" event={"ID":"09f12a70-d45e-4340-bdaf-c61d3af3d130","Type":"ContainerStarted","Data":"731367e3ff00edc8d8de0aa2d006306f5e5d345c67fbfd43f085d64d00a3745a"} Nov 01 16:30:11.977148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.977049 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" event={"ID":"09f12a70-d45e-4340-bdaf-c61d3af3d130","Type":"ContainerStarted","Data":"6898ecaa60dc086209a460636cd913e63883c4c7d3f9627cc1fc4369ea087e32"} Nov 01 16:30:11.977148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:11.977115 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:12.542657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:12.542625 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:13.424511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:13.424477 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:14.159131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:14.159093 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:14.451537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:14.451458 4573 scope.go:122] "RemoveContainer" containerID="49dec7accf92d4af2e2943a89b03e2b6390af0e0f87f5d7ba3c38a46c52b9660" Nov 01 16:30:14.979475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:14.979443 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:15.003079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:15.003037 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" event={"ID":"bb0a40d1-ad36-416e-94d5-3ca525262685","Type":"ContainerStarted","Data":"248f2589a6927cbcec52991871af9e59e750b81776a86daf9a51313b9f685525"} Nov 01 16:30:15.018815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:15.018770 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" podStartSLOduration=4.018743488 podStartE2EDuration="4.018743488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:12.001106929 +0000 UTC m=+2549.704272866" watchObservedRunningTime="2026-11-01 16:30:15.018743488 +0000 UTC m=+2552.721909496" Nov 01 16:30:15.451067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:15.451025 4573 scope.go:122] "RemoveContainer" containerID="2a920de98c9855b2facea6e58a516bb7f2f9e76d3cf47424ef1e54e11a4e3799" Nov 01 16:30:15.451272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:15.451191 4573 scope.go:122] "RemoveContainer" containerID="36301111b007ec03b7d6b056c95ec99eee6208ad78ce3d586c0ffc018fada3d7" Nov 01 16:30:15.451363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:15.451336 4573 scope.go:122] "RemoveContainer" containerID="53d84369a77ee43c49d319f7d90a1e3d3a0a64f9439dc3a63ba6065199e172a9" Nov 01 16:30:15.622681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:15.622651 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:16.016305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.016268 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" event={"ID":"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d","Type":"ContainerStarted","Data":"f9d93fbf85ed326cb8db64cbabb0ee39f0c750dafc057b1aa3509f0200c4490c"} Nov 01 16:30:16.018341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.018274 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" event={"ID":"7812173e-6df9-4f18-a704-b316ca57c568","Type":"ContainerStarted","Data":"dbb2f1b11a2cffa14620ce4112b17769d7f0d202be8d237112ebd6f3b27536e8"} Nov 01 16:30:16.020254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.020229 4573 generic.go:356] "Generic (PLEG): container finished" podID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerID="248f2589a6927cbcec52991871af9e59e750b81776a86daf9a51313b9f685525" exitCode=60 Nov 01 16:30:16.020390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.020301 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" event={"ID":"bb0a40d1-ad36-416e-94d5-3ca525262685","Type":"ContainerDied","Data":"248f2589a6927cbcec52991871af9e59e750b81776a86daf9a51313b9f685525"} Nov 01 16:30:16.020390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.020371 4573 scope.go:122] "RemoveContainer" containerID="49dec7accf92d4af2e2943a89b03e2b6390af0e0f87f5d7ba3c38a46c52b9660" Nov 01 16:30:16.020771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.020729 4573 scope.go:122] "RemoveContainer" containerID="248f2589a6927cbcec52991871af9e59e750b81776a86daf9a51313b9f685525" Nov 01 16:30:16.021137 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:16.021100 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b_odh-ai-gateway-infra(bb0a40d1-ad36-416e-94d5-3ca525262685)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" Nov 01 16:30:16.022344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.022324 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" event={"ID":"1c48c30a-0215-4276-95a8-f1ebced71602","Type":"ContainerStarted","Data":"00dfeb343abd69ea99dfc795f082898bd807c9ac43f8419e609bc5b42c7a66ec"} Nov 01 16:30:16.029174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.029121 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b"] Nov 01 16:30:16.388522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.388470 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:16.459792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.459762 4573 scope.go:122] "RemoveContainer" containerID="6e127073cacb2fe9aeb021a8924cf2afa01a3bb23de63be6ec9dc0f97bb9c572" Nov 01 16:30:16.847673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.847636 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:16.945105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:16.945025 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:17.120777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.117003 4573 generic.go:356] "Generic (PLEG): container finished" podID="7812173e-6df9-4f18-a704-b316ca57c568" containerID="dbb2f1b11a2cffa14620ce4112b17769d7f0d202be8d237112ebd6f3b27536e8" exitCode=60 Nov 01 16:30:17.120777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.117170 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" event={"ID":"7812173e-6df9-4f18-a704-b316ca57c568","Type":"ContainerDied","Data":"dbb2f1b11a2cffa14620ce4112b17769d7f0d202be8d237112ebd6f3b27536e8"} Nov 01 16:30:17.120777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.117257 4573 scope.go:122] "RemoveContainer" containerID="2a920de98c9855b2facea6e58a516bb7f2f9e76d3cf47424ef1e54e11a4e3799" Nov 01 16:30:17.120777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.117785 4573 scope.go:122] "RemoveContainer" containerID="dbb2f1b11a2cffa14620ce4112b17769d7f0d202be8d237112ebd6f3b27536e8" Nov 01 16:30:17.120777 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:17.118191 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44_odh-ai-gateway-infra(7812173e-6df9-4f18-a704-b316ca57c568)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" podUID="7812173e-6df9-4f18-a704-b316ca57c568" Nov 01 16:30:17.137780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.136529 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w"] Nov 01 16:30:17.141799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.141455 4573 generic.go:356] "Generic (PLEG): container finished" podID="1c48c30a-0215-4276-95a8-f1ebced71602" containerID="00dfeb343abd69ea99dfc795f082898bd807c9ac43f8419e609bc5b42c7a66ec" exitCode=60 Nov 01 16:30:17.141799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.141690 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" event={"ID":"1c48c30a-0215-4276-95a8-f1ebced71602","Type":"ContainerDied","Data":"00dfeb343abd69ea99dfc795f082898bd807c9ac43f8419e609bc5b42c7a66ec"} Nov 01 16:30:17.147040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.142182 4573 scope.go:122] "RemoveContainer" containerID="00dfeb343abd69ea99dfc795f082898bd807c9ac43f8419e609bc5b42c7a66ec" Nov 01 16:30:17.147040 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:17.142613 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg_odh-ai-gateway-infra(1c48c30a-0215-4276-95a8-f1ebced71602)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" Nov 01 16:30:17.147040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.146570 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" event={"ID":"93f3372f-7731-4b2d-842c-3be1a6c71855","Type":"ContainerStarted","Data":"3565b603ba8fccc39dfef13b56fb24e1b2ee0eea9e2320e1212c72fd39de5088"} Nov 01 16:30:17.168161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.150900 4573 generic.go:356] "Generic (PLEG): container finished" podID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerID="f9d93fbf85ed326cb8db64cbabb0ee39f0c750dafc057b1aa3509f0200c4490c" exitCode=60 Nov 01 16:30:17.168161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.150968 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" event={"ID":"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d","Type":"ContainerDied","Data":"f9d93fbf85ed326cb8db64cbabb0ee39f0c750dafc057b1aa3509f0200c4490c"} Nov 01 16:30:17.189280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.188227 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg"] Nov 01 16:30:17.199238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.199079 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44"] Nov 01 16:30:17.210449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.210272 4573 scope.go:122] "RemoveContainer" containerID="53d84369a77ee43c49d319f7d90a1e3d3a0a64f9439dc3a63ba6065199e172a9" Nov 01 16:30:17.236479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.233007 4573 scope.go:122] "RemoveContainer" containerID="36301111b007ec03b7d6b056c95ec99eee6208ad78ce3d586c0ffc018fada3d7" Nov 01 16:30:17.289673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.288440 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:17.461786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.450220 4573 scope.go:122] "RemoveContainer" containerID="fa522e39c4d4cc84aa7513fa9782d4481e4a440043f8279b0afade51596c244c" Nov 01 16:30:17.461786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.451035 4573 scope.go:122] "RemoveContainer" containerID="9c43b4e8cf8c4a2a2a1a50016048c756ac6471f2a559dee83cbf56e5093a8a31" Nov 01 16:30:17.462947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.462053 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:17.496629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.496599 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0a40d1-ad36-416e-94d5-3ca525262685-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a40d1-ad36-416e-94d5-3ca525262685-maas-api-service-ca\") pod \"bb0a40d1-ad36-416e-94d5-3ca525262685\" (UID: \"bb0a40d1-ad36-416e-94d5-3ca525262685\") " Nov 01 16:30:17.496789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.496698 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0a40d1-ad36-416e-94d5-3ca525262685-kube-api-access-b268r\" (UniqueName: \"kubernetes.io/projected/bb0a40d1-ad36-416e-94d5-3ca525262685-kube-api-access-b268r\") pod \"bb0a40d1-ad36-416e-94d5-3ca525262685\" (UID: \"bb0a40d1-ad36-416e-94d5-3ca525262685\") " Nov 01 16:30:17.498211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.498183 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0a40d1-ad36-416e-94d5-3ca525262685-maas-api-service-ca" pod "bb0a40d1-ad36-416e-94d5-3ca525262685" (UID: "bb0a40d1-ad36-416e-94d5-3ca525262685"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:17.500940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.500910 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0a40d1-ad36-416e-94d5-3ca525262685-kube-api-access-b268r" pod "bb0a40d1-ad36-416e-94d5-3ca525262685" (UID: "bb0a40d1-ad36-416e-94d5-3ca525262685"). InnerVolumeSpecName "kube-api-access-b268r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:17.504984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.504960 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:17.598531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.598504 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-maas-api-service-ca\") pod \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\" (UID: \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\") " Nov 01 16:30:17.598697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.598605 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-kube-api-access-6jzbq\") pod \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\" (UID: \"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d\") " Nov 01 16:30:17.598876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.598856 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-maas-api-service-ca" pod "e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" (UID: "e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:17.599064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.599044 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb0a40d1-ad36-416e-94d5-3ca525262685-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:17.599064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.599063 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b268r\" (UniqueName: \"kubernetes.io/projected/bb0a40d1-ad36-416e-94d5-3ca525262685-kube-api-access-b268r\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:17.599212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.599074 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:17.664889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.664623 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-kube-api-access-6jzbq" pod "e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" (UID: "e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d"). InnerVolumeSpecName "kube-api-access-6jzbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:17.700579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:17.700550 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jzbq\" (UniqueName: \"kubernetes.io/projected/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d-kube-api-access-6jzbq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.163185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.163156 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" Nov 01 16:30:18.163646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.163156 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b" event={"ID":"bb0a40d1-ad36-416e-94d5-3ca525262685","Type":"ContainerDied","Data":"444c72bdfd8ab585e681d6a7454b33b72e56a9c3149551a4efb2a6aab3ccb807"} Nov 01 16:30:18.163646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.163320 4573 scope.go:122] "RemoveContainer" containerID="248f2589a6927cbcec52991871af9e59e750b81776a86daf9a51313b9f685525" Nov 01 16:30:18.168615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.168588 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" event={"ID":"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6","Type":"ContainerStarted","Data":"5cd5ab681469fdca9322cc2b58a34c84e2ff3c435dc9691afafd7f9628034222"} Nov 01 16:30:18.171671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.171000 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" event={"ID":"3d1343be-17df-4d00-a24b-37426a338c89","Type":"ContainerStarted","Data":"a0efb0c78471c9e91757aeefec632183c4b90c775d787317086805d62b0609a9"} Nov 01 16:30:18.173621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.173432 4573 generic.go:356] "Generic (PLEG): container finished" podID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerID="3565b603ba8fccc39dfef13b56fb24e1b2ee0eea9e2320e1212c72fd39de5088" exitCode=60 Nov 01 16:30:18.173621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.173588 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" event={"ID":"93f3372f-7731-4b2d-842c-3be1a6c71855","Type":"ContainerDied","Data":"3565b603ba8fccc39dfef13b56fb24e1b2ee0eea9e2320e1212c72fd39de5088"} Nov 01 16:30:18.174032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.174011 4573 scope.go:122] "RemoveContainer" containerID="3565b603ba8fccc39dfef13b56fb24e1b2ee0eea9e2320e1212c72fd39de5088" Nov 01 16:30:18.174366 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:18.174341 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p_odh-ai-gateway-infra(93f3372f-7731-4b2d-842c-3be1a6c71855)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" Nov 01 16:30:18.178838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.178801 4573 scope.go:122] "RemoveContainer" containerID="6e127073cacb2fe9aeb021a8924cf2afa01a3bb23de63be6ec9dc0f97bb9c572" Nov 01 16:30:18.180223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.180163 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" event={"ID":"e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d","Type":"ContainerDied","Data":"786bd3a2d5d0c092a76f7118009f36267055b7ea2fa46384f4e47076ab2eebfc"} Nov 01 16:30:18.180403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.180378 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w" Nov 01 16:30:18.211342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.211311 4573 scope.go:122] "RemoveContainer" containerID="f9d93fbf85ed326cb8db64cbabb0ee39f0c750dafc057b1aa3509f0200c4490c" Nov 01 16:30:18.228812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.228772 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p"] Nov 01 16:30:18.250841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.250812 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b"] Nov 01 16:30:18.257023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.255447 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-734859-29892510-sq45b"] Nov 01 16:30:18.278712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.278680 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:18.313835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.313800 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w"] Nov 01 16:30:18.318454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.318429 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-dfff59-29892510-6ws5w"] Nov 01 16:30:18.450101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.450076 4573 scope.go:122] "RemoveContainer" containerID="f99b9e1c21e09bed99979f8d1074404e1a4bbdbae2e9c38563b91385a80917ed" Nov 01 16:30:18.453128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.453106 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:18.459301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.459283 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:18.463502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.463472 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" path="/var/lib/kubelet/pods/bb0a40d1-ad36-416e-94d5-3ca525262685/volumes" Nov 01 16:30:18.464069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.464003 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" path="/var/lib/kubelet/pods/e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d/volumes" Nov 01 16:30:18.517315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.517282 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c48c30a-0215-4276-95a8-f1ebced71602-kube-api-access-whp8z\" (UniqueName: \"kubernetes.io/projected/1c48c30a-0215-4276-95a8-f1ebced71602-kube-api-access-whp8z\") pod \"1c48c30a-0215-4276-95a8-f1ebced71602\" (UID: \"1c48c30a-0215-4276-95a8-f1ebced71602\") " Nov 01 16:30:18.517485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.517465 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c48c30a-0215-4276-95a8-f1ebced71602-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c48c30a-0215-4276-95a8-f1ebced71602-maas-api-service-ca\") pod \"1c48c30a-0215-4276-95a8-f1ebced71602\" (UID: \"1c48c30a-0215-4276-95a8-f1ebced71602\") " Nov 01 16:30:18.517679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.517652 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7812173e-6df9-4f18-a704-b316ca57c568-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812173e-6df9-4f18-a704-b316ca57c568-maas-api-service-ca\") pod \"7812173e-6df9-4f18-a704-b316ca57c568\" (UID: \"7812173e-6df9-4f18-a704-b316ca57c568\") " Nov 01 16:30:18.517922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.517701 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7812173e-6df9-4f18-a704-b316ca57c568-kube-api-access-272bz\" (UniqueName: \"kubernetes.io/projected/7812173e-6df9-4f18-a704-b316ca57c568-kube-api-access-272bz\") pod \"7812173e-6df9-4f18-a704-b316ca57c568\" (UID: \"7812173e-6df9-4f18-a704-b316ca57c568\") " Nov 01 16:30:18.518037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.518015 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c48c30a-0215-4276-95a8-f1ebced71602-maas-api-service-ca" pod "1c48c30a-0215-4276-95a8-f1ebced71602" (UID: "1c48c30a-0215-4276-95a8-f1ebced71602"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:18.518539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.518511 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c48c30a-0215-4276-95a8-f1ebced71602-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.520604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.520578 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7812173e-6df9-4f18-a704-b316ca57c568-maas-api-service-ca" pod "7812173e-6df9-4f18-a704-b316ca57c568" (UID: "7812173e-6df9-4f18-a704-b316ca57c568"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:18.521302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.521277 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c48c30a-0215-4276-95a8-f1ebced71602-kube-api-access-whp8z" pod "1c48c30a-0215-4276-95a8-f1ebced71602" (UID: "1c48c30a-0215-4276-95a8-f1ebced71602"). InnerVolumeSpecName "kube-api-access-whp8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:18.521414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.521315 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7812173e-6df9-4f18-a704-b316ca57c568-kube-api-access-272bz" pod "7812173e-6df9-4f18-a704-b316ca57c568" (UID: "7812173e-6df9-4f18-a704-b316ca57c568"). InnerVolumeSpecName "kube-api-access-272bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:18.619883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.619849 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whp8z\" (UniqueName: \"kubernetes.io/projected/1c48c30a-0215-4276-95a8-f1ebced71602-kube-api-access-whp8z\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.619883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.619878 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7812173e-6df9-4f18-a704-b316ca57c568-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:18.619883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:18.619888 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-272bz\" (UniqueName: \"kubernetes.io/projected/7812173e-6df9-4f18-a704-b316ca57c568-kube-api-access-272bz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:19.169535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.169503 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:19.190916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.190881 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" event={"ID":"1c48c30a-0215-4276-95a8-f1ebced71602","Type":"ContainerDied","Data":"89b65450653a027957c3845530e27afa056e4853f4f0ab83d75a64a8d8fa2466"} Nov 01 16:30:19.190916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.190909 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg" Nov 01 16:30:19.191188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.190972 4573 scope.go:122] "RemoveContainer" containerID="00dfeb343abd69ea99dfc795f082898bd807c9ac43f8419e609bc5b42c7a66ec" Nov 01 16:30:19.193190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.193165 4573 generic.go:356] "Generic (PLEG): container finished" podID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerID="5cd5ab681469fdca9322cc2b58a34c84e2ff3c435dc9691afafd7f9628034222" exitCode=60 Nov 01 16:30:19.193471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.193442 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" event={"ID":"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6","Type":"ContainerDied","Data":"5cd5ab681469fdca9322cc2b58a34c84e2ff3c435dc9691afafd7f9628034222"} Nov 01 16:30:19.194056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.193953 4573 scope.go:122] "RemoveContainer" containerID="5cd5ab681469fdca9322cc2b58a34c84e2ff3c435dc9691afafd7f9628034222" Nov 01 16:30:19.194377 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:19.194349 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94_odh-ai-gateway-infra(3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" Nov 01 16:30:19.196011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.195991 4573 generic.go:356] "Generic (PLEG): container finished" podID="3d1343be-17df-4d00-a24b-37426a338c89" containerID="a0efb0c78471c9e91757aeefec632183c4b90c775d787317086805d62b0609a9" exitCode=60 Nov 01 16:30:19.196127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.196066 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" event={"ID":"3d1343be-17df-4d00-a24b-37426a338c89","Type":"ContainerDied","Data":"a0efb0c78471c9e91757aeefec632183c4b90c775d787317086805d62b0609a9"} Nov 01 16:30:19.196487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.196454 4573 scope.go:122] "RemoveContainer" containerID="a0efb0c78471c9e91757aeefec632183c4b90c775d787317086805d62b0609a9" Nov 01 16:30:19.196801 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:19.196776 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58_odh-ai-gateway-infra(3d1343be-17df-4d00-a24b-37426a338c89)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" podUID="3d1343be-17df-4d00-a24b-37426a338c89" Nov 01 16:30:19.201466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.201423 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" event={"ID":"693232a8-19dd-44b9-ab5e-b073b3dbd82c","Type":"ContainerStarted","Data":"3676ff921ace3b5a602fec726f40d7556c940978435dc2e21da89416a29cdcf6"} Nov 01 16:30:19.203110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.203092 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" event={"ID":"7812173e-6df9-4f18-a704-b316ca57c568","Type":"ContainerDied","Data":"6b520c76be291fdd17e9cf4ad3ef2b21d69e48882313a484ba9deeeff6d6c1cb"} Nov 01 16:30:19.203214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.203116 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44" Nov 01 16:30:19.205438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.205418 4573 scope.go:122] "RemoveContainer" containerID="9c43b4e8cf8c4a2a2a1a50016048c756ac6471f2a559dee83cbf56e5093a8a31" Nov 01 16:30:19.272826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.266487 4573 scope.go:122] "RemoveContainer" containerID="fa522e39c4d4cc84aa7513fa9782d4481e4a440043f8279b0afade51596c244c" Nov 01 16:30:19.286185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.286153 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94"] Nov 01 16:30:19.297209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.297184 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58"] Nov 01 16:30:19.303305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.303256 4573 scope.go:122] "RemoveContainer" containerID="dbb2f1b11a2cffa14620ce4112b17769d7f0d202be8d237112ebd6f3b27536e8" Nov 01 16:30:19.345791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.344223 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg"] Nov 01 16:30:19.352299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.352270 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a9a967-29892510-jgkdg"] Nov 01 16:30:19.377643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.377603 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44"] Nov 01 16:30:19.386384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.386337 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bd831a-29892510-9wl44"] Nov 01 16:30:19.441141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.441082 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:19.546833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.546803 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f3372f-7731-4b2d-842c-3be1a6c71855-kube-api-access-wnd48\" (UniqueName: \"kubernetes.io/projected/93f3372f-7731-4b2d-842c-3be1a6c71855-kube-api-access-wnd48\") pod \"93f3372f-7731-4b2d-842c-3be1a6c71855\" (UID: \"93f3372f-7731-4b2d-842c-3be1a6c71855\") " Nov 01 16:30:19.546986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.546881 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f3372f-7731-4b2d-842c-3be1a6c71855-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f3372f-7731-4b2d-842c-3be1a6c71855-maas-api-service-ca\") pod \"93f3372f-7731-4b2d-842c-3be1a6c71855\" (UID: \"93f3372f-7731-4b2d-842c-3be1a6c71855\") " Nov 01 16:30:19.547194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.547170 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f3372f-7731-4b2d-842c-3be1a6c71855-maas-api-service-ca" pod "93f3372f-7731-4b2d-842c-3be1a6c71855" (UID: "93f3372f-7731-4b2d-842c-3be1a6c71855"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:19.547377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.547361 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93f3372f-7731-4b2d-842c-3be1a6c71855-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:19.548960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.548940 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f3372f-7731-4b2d-842c-3be1a6c71855-kube-api-access-wnd48" pod "93f3372f-7731-4b2d-842c-3be1a6c71855" (UID: "93f3372f-7731-4b2d-842c-3be1a6c71855"). InnerVolumeSpecName "kube-api-access-wnd48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:19.649226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.649196 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnd48\" (UniqueName: \"kubernetes.io/projected/93f3372f-7731-4b2d-842c-3be1a6c71855-kube-api-access-wnd48\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:19.987375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:19.987339 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:20.213486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.213456 4573 generic.go:356] "Generic (PLEG): container finished" podID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerID="3676ff921ace3b5a602fec726f40d7556c940978435dc2e21da89416a29cdcf6" exitCode=60 Nov 01 16:30:20.213979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.213528 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" event={"ID":"693232a8-19dd-44b9-ab5e-b073b3dbd82c","Type":"ContainerDied","Data":"3676ff921ace3b5a602fec726f40d7556c940978435dc2e21da89416a29cdcf6"} Nov 01 16:30:20.213979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.213596 4573 scope.go:122] "RemoveContainer" containerID="f99b9e1c21e09bed99979f8d1074404e1a4bbdbae2e9c38563b91385a80917ed" Nov 01 16:30:20.214092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.214040 4573 scope.go:122] "RemoveContainer" containerID="3676ff921ace3b5a602fec726f40d7556c940978435dc2e21da89416a29cdcf6" Nov 01 16:30:20.214463 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:20.214428 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb_odh-ai-gateway-infra(693232a8-19dd-44b9-ab5e-b073b3dbd82c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" Nov 01 16:30:20.221425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.221405 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" event={"ID":"93f3372f-7731-4b2d-842c-3be1a6c71855","Type":"ContainerDied","Data":"4557c2e1398148625c451ede341b79316c9e5e4fe1d183ee866be74c05f8fbf5"} Nov 01 16:30:20.221554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.221428 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p" Nov 01 16:30:20.235598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.235435 4573 scope.go:122] "RemoveContainer" containerID="3565b603ba8fccc39dfef13b56fb24e1b2ee0eea9e2320e1212c72fd39de5088" Nov 01 16:30:20.273019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.272981 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb"] Nov 01 16:30:20.296036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.296004 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p"] Nov 01 16:30:20.315321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.315296 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9a482b-29892510-cz24p"] Nov 01 16:30:20.454243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.452405 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:20.464864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.464830 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" path="/var/lib/kubelet/pods/1c48c30a-0215-4276-95a8-f1ebced71602/volumes" Nov 01 16:30:20.465453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.465434 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7812173e-6df9-4f18-a704-b316ca57c568" path="/var/lib/kubelet/pods/7812173e-6df9-4f18-a704-b316ca57c568/volumes" Nov 01 16:30:20.466047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.466025 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" path="/var/lib/kubelet/pods/93f3372f-7731-4b2d-842c-3be1a6c71855/volumes" Nov 01 16:30:20.510910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.510878 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:20.564871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.564841 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1343be-17df-4d00-a24b-37426a338c89-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1343be-17df-4d00-a24b-37426a338c89-maas-api-service-ca\") pod \"3d1343be-17df-4d00-a24b-37426a338c89\" (UID: \"3d1343be-17df-4d00-a24b-37426a338c89\") " Nov 01 16:30:20.565029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.564946 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1343be-17df-4d00-a24b-37426a338c89-kube-api-access-4vndv\" (UniqueName: \"kubernetes.io/projected/3d1343be-17df-4d00-a24b-37426a338c89-kube-api-access-4vndv\") pod \"3d1343be-17df-4d00-a24b-37426a338c89\" (UID: \"3d1343be-17df-4d00-a24b-37426a338c89\") " Nov 01 16:30:20.565266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.565237 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1343be-17df-4d00-a24b-37426a338c89-maas-api-service-ca" pod "3d1343be-17df-4d00-a24b-37426a338c89" (UID: "3d1343be-17df-4d00-a24b-37426a338c89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:20.565954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.565923 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d1343be-17df-4d00-a24b-37426a338c89-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.567208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.567179 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1343be-17df-4d00-a24b-37426a338c89-kube-api-access-4vndv" pod "3d1343be-17df-4d00-a24b-37426a338c89" (UID: "3d1343be-17df-4d00-a24b-37426a338c89"). InnerVolumeSpecName "kube-api-access-4vndv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:20.667056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.667031 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-maas-api-service-ca\") pod \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\" (UID: \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\") " Nov 01 16:30:20.667234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.667084 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-kube-api-access-djxwn\" (UniqueName: \"kubernetes.io/projected/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-kube-api-access-djxwn\") pod \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\" (UID: \"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6\") " Nov 01 16:30:20.667467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.667441 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-maas-api-service-ca" pod "3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" (UID: "3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:20.667825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.667802 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.667825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.667825 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vndv\" (UniqueName: \"kubernetes.io/projected/3d1343be-17df-4d00-a24b-37426a338c89-kube-api-access-4vndv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.669846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.669816 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-kube-api-access-djxwn" pod "3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" (UID: "3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6"). InnerVolumeSpecName "kube-api-access-djxwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:20.769487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.769410 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djxwn\" (UniqueName: \"kubernetes.io/projected/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6-kube-api-access-djxwn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:20.775296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:20.775269 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:21.233878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.233855 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" Nov 01 16:30:21.234285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.233895 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94" event={"ID":"3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6","Type":"ContainerDied","Data":"677055584bf0c4ac1ce42f9f9a4dc5f95a421a2a193cba4b6334f0dddb8a3116"} Nov 01 16:30:21.234285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.233965 4573 scope.go:122] "RemoveContainer" containerID="5cd5ab681469fdca9322cc2b58a34c84e2ff3c435dc9691afafd7f9628034222" Nov 01 16:30:21.235734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.235712 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" Nov 01 16:30:21.235878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.235745 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58" event={"ID":"3d1343be-17df-4d00-a24b-37426a338c89","Type":"ContainerDied","Data":"7a5c22af2eafd166485de46a5a15548127dc7b44e1a94877303dce54d1048622"} Nov 01 16:30:21.250354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.250326 4573 scope.go:122] "RemoveContainer" containerID="a0efb0c78471c9e91757aeefec632183c4b90c775d787317086805d62b0609a9" Nov 01 16:30:21.272095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.272064 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94"] Nov 01 16:30:21.279242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.278947 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-e1e630-29892510-xtf94"] Nov 01 16:30:21.294342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.294312 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58"] Nov 01 16:30:21.298247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.298219 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-164db6-29892510-wsg58"] Nov 01 16:30:21.543817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.538670 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:21.601217 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.601182 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/693232a8-19dd-44b9-ab5e-b073b3dbd82c-kube-api-access-478vv\" (UniqueName: \"kubernetes.io/projected/693232a8-19dd-44b9-ab5e-b073b3dbd82c-kube-api-access-478vv\") pod \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\" (UID: \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\") " Nov 01 16:30:21.601391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.601313 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/693232a8-19dd-44b9-ab5e-b073b3dbd82c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/693232a8-19dd-44b9-ab5e-b073b3dbd82c-maas-api-service-ca\") pod \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\" (UID: \"693232a8-19dd-44b9-ab5e-b073b3dbd82c\") " Nov 01 16:30:21.601718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.601686 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/693232a8-19dd-44b9-ab5e-b073b3dbd82c-maas-api-service-ca" pod "693232a8-19dd-44b9-ab5e-b073b3dbd82c" (UID: "693232a8-19dd-44b9-ab5e-b073b3dbd82c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:21.601973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.601939 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/693232a8-19dd-44b9-ab5e-b073b3dbd82c-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:21.604080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.604048 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/693232a8-19dd-44b9-ab5e-b073b3dbd82c-kube-api-access-478vv" pod "693232a8-19dd-44b9-ab5e-b073b3dbd82c" (UID: "693232a8-19dd-44b9-ab5e-b073b3dbd82c"). InnerVolumeSpecName "kube-api-access-478vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:21.702928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.702891 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-478vv\" (UniqueName: \"kubernetes.io/projected/693232a8-19dd-44b9-ab5e-b073b3dbd82c-kube-api-access-478vv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:21.712816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:21.712786 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:22.248078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.248051 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" Nov 01 16:30:22.248078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.248060 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb" event={"ID":"693232a8-19dd-44b9-ab5e-b073b3dbd82c","Type":"ContainerDied","Data":"df81b7a04519bd5e2d18113a453d9c2df71289647f72e649ee1d3d55d21ae605"} Nov 01 16:30:22.248595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.248120 4573 scope.go:122] "RemoveContainer" containerID="3676ff921ace3b5a602fec726f40d7556c940978435dc2e21da89416a29cdcf6" Nov 01 16:30:22.277473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.277444 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb"] Nov 01 16:30:22.284807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.283001 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9f4dc9-29892510-mwxqb"] Nov 01 16:30:22.460546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.460513 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1343be-17df-4d00-a24b-37426a338c89" path="/var/lib/kubelet/pods/3d1343be-17df-4d00-a24b-37426a338c89/volumes" Nov 01 16:30:22.461041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.461026 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" path="/var/lib/kubelet/pods/3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6/volumes" Nov 01 16:30:22.461424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.461411 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" path="/var/lib/kubelet/pods/693232a8-19dd-44b9-ab5e-b073b3dbd82c/volumes" Nov 01 16:30:22.549289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.549146 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:22.998342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:22.998310 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:23.303246 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:23.303168 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c"] Nov 01 16:30:23.328496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:23.328453 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv"] Nov 01 16:30:23.356523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:23.356479 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:23.456432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:23.456400 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:23.624206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:23.624165 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:24.066924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:24.066839 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:24.365262 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:24.365230 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:30:24.860826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:24.860788 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:24.977683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:24.977651 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:25.058156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:25.058106 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:25.761942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:25.761906 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:26.444974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:26.444940 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:27.189520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:27.189487 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:27.875146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:27.875114 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:28.023406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.023367 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q"] Nov 01 16:30:28.027708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.027686 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce8f835d-dlb4q"] Nov 01 16:30:28.303943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.303861 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" podUID="839d2919-d17c-4e19-bdba-dcbb222ba1d1" containerName="payload-processing" containerID="cri-o://4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7" gracePeriod=25 Nov 01 16:30:28.329809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.329777 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" podUID="7e74a675-d484-496a-a223-1d03ca77f424" containerName="payload-pre-processing" containerID="cri-o://9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce" gracePeriod=25 Nov 01 16:30:28.460738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.460705 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0c76cc-4a3b-4205-99ff-850da03881b6" path="/var/lib/kubelet/pods/ea0c76cc-4a3b-4205-99ff-850da03881b6/volumes" Nov 01 16:30:28.642664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.642634 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:30:28.645488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.645468 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:30:28.681944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.681851 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:28.791405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.791376 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e74a675-d484-496a-a223-1d03ca77f424-kube-api-access-nv4ts\" (UniqueName: \"kubernetes.io/projected/7e74a675-d484-496a-a223-1d03ca77f424-kube-api-access-nv4ts\") pod \"7e74a675-d484-496a-a223-1d03ca77f424\" (UID: \"7e74a675-d484-496a-a223-1d03ca77f424\") " Nov 01 16:30:28.791564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.791533 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e74a675-d484-496a-a223-1d03ca77f424-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74a675-d484-496a-a223-1d03ca77f424-plugins-config-volume\") pod \"7e74a675-d484-496a-a223-1d03ca77f424\" (UID: \"7e74a675-d484-496a-a223-1d03ca77f424\") " Nov 01 16:30:28.791608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.791579 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839d2919-d17c-4e19-bdba-dcbb222ba1d1-kube-api-access-zr7kp\" (UniqueName: \"kubernetes.io/projected/839d2919-d17c-4e19-bdba-dcbb222ba1d1-kube-api-access-zr7kp\") pod \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\" (UID: \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\") " Nov 01 16:30:28.791667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.791657 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839d2919-d17c-4e19-bdba-dcbb222ba1d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839d2919-d17c-4e19-bdba-dcbb222ba1d1-plugins-config-volume\") pod \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\" (UID: \"839d2919-d17c-4e19-bdba-dcbb222ba1d1\") " Nov 01 16:30:28.791980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.791930 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e74a675-d484-496a-a223-1d03ca77f424-plugins-config-volume" pod "7e74a675-d484-496a-a223-1d03ca77f424" (UID: "7e74a675-d484-496a-a223-1d03ca77f424"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:28.792180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.792152 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839d2919-d17c-4e19-bdba-dcbb222ba1d1-plugins-config-volume" pod "839d2919-d17c-4e19-bdba-dcbb222ba1d1" (UID: "839d2919-d17c-4e19-bdba-dcbb222ba1d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:28.792292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.792237 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e74a675-d484-496a-a223-1d03ca77f424-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:28.793611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.793591 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e74a675-d484-496a-a223-1d03ca77f424-kube-api-access-nv4ts" pod "7e74a675-d484-496a-a223-1d03ca77f424" (UID: "7e74a675-d484-496a-a223-1d03ca77f424"). InnerVolumeSpecName "kube-api-access-nv4ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:28.793724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.793706 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839d2919-d17c-4e19-bdba-dcbb222ba1d1-kube-api-access-zr7kp" pod "839d2919-d17c-4e19-bdba-dcbb222ba1d1" (UID: "839d2919-d17c-4e19-bdba-dcbb222ba1d1"). InnerVolumeSpecName "kube-api-access-zr7kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:28.893903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.893878 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7kp\" (UniqueName: \"kubernetes.io/projected/839d2919-d17c-4e19-bdba-dcbb222ba1d1-kube-api-access-zr7kp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:28.893903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.893903 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839d2919-d17c-4e19-bdba-dcbb222ba1d1-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:28.894093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:28.893913 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv4ts\" (UniqueName: \"kubernetes.io/projected/7e74a675-d484-496a-a223-1d03ca77f424-kube-api-access-nv4ts\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:29.229971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.229875 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:29.314964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.314927 4573 generic.go:356] "Generic (PLEG): container finished" podID="7e74a675-d484-496a-a223-1d03ca77f424" containerID="9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce" exitCode=0 Nov 01 16:30:29.315387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.314991 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" Nov 01 16:30:29.315387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.315011 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" event={"ID":"7e74a675-d484-496a-a223-1d03ca77f424","Type":"ContainerDied","Data":"9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce"} Nov 01 16:30:29.315387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.315103 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv" event={"ID":"7e74a675-d484-496a-a223-1d03ca77f424","Type":"ContainerDied","Data":"affe450ee2433e07d45a49798a9087bb969de57597dd21d6ca144fcf975b066a"} Nov 01 16:30:29.315387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.315131 4573 scope.go:122] "RemoveContainer" containerID="9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce" Nov 01 16:30:29.316873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.316848 4573 generic.go:356] "Generic (PLEG): container finished" podID="839d2919-d17c-4e19-bdba-dcbb222ba1d1" containerID="4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7" exitCode=0 Nov 01 16:30:29.316997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.316918 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" event={"ID":"839d2919-d17c-4e19-bdba-dcbb222ba1d1","Type":"ContainerDied","Data":"4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7"} Nov 01 16:30:29.316997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.316958 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" event={"ID":"839d2919-d17c-4e19-bdba-dcbb222ba1d1","Type":"ContainerDied","Data":"0b5b6b7f689e2bc10ea996cf6452e026a5c4a53eff8892e8433cabeb01e60004"} Nov 01 16:30:29.316997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.316921 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c" Nov 01 16:30:29.334218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.334194 4573 scope.go:122] "RemoveContainer" containerID="9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce" Nov 01 16:30:29.335233 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:29.335208 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce\": container with ID starting with 9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce not found: ID does not exist" containerID="9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce" Nov 01 16:30:29.335326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.335248 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce"} err="failed to get container status \"9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce\": rpc error: code = NotFound desc = could not find container \"9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce\": container with ID starting with 9de2c0c1c76e8ea0de83de5b97d2233276235196401e5934a5e1798b7878c2ce not found: ID does not exist" Nov 01 16:30:29.335326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.335270 4573 scope.go:122] "RemoveContainer" containerID="4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7" Nov 01 16:30:29.346157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.346137 4573 scope.go:122] "RemoveContainer" containerID="4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7" Nov 01 16:30:29.346442 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:29.346423 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7\": container with ID starting with 4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7 not found: ID does not exist" containerID="4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7" Nov 01 16:30:29.346493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.346450 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7"} err="failed to get container status \"4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7\": rpc error: code = NotFound desc = could not find container \"4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7\": container with ID starting with 4f8d9bd449f5e45644bc2c10f20441a501f7f77f1d96b62272cbd449a25b5eb7 not found: ID does not exist" Nov 01 16:30:29.348168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.348152 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c"] Nov 01 16:30:29.353856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.353836 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-028c46-675df4f67f-8l65c"] Nov 01 16:30:29.363374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.363347 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv"] Nov 01 16:30:29.367656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.367635 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-568c99d987-m48cv"] Nov 01 16:30:29.385742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.385718 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:29.472773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.464187 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:29.776821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:29.776738 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:30.148611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:30.148577 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:30.460288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:30.460198 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e74a675-d484-496a-a223-1d03ca77f424" path="/var/lib/kubelet/pods/7e74a675-d484-496a-a223-1d03ca77f424/volumes" Nov 01 16:30:30.460777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:30.460736 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839d2919-d17c-4e19-bdba-dcbb222ba1d1" path="/var/lib/kubelet/pods/839d2919-d17c-4e19-bdba-dcbb222ba1d1/volumes" Nov 01 16:30:30.573215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:30.573178 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:31.263185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:31.263145 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:31.950645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:31.950597 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:32.311472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:32.311397 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:32.602950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:32.602867 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:33.471012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:33.470975 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:34.175710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.175674 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:34.671040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.671002 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv"] Nov 01 16:30:34.671523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.671213 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" podUID="719e9e9e-90ae-4178-b113-96166fdeb425" containerName="maas-api" containerID="cri-o://9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5" gracePeriod=30 Nov 01 16:30:34.701175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.701138 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:34.746025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.745992 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm"] Nov 01 16:30:34.753208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.753180 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm"] Nov 01 16:30:34.755091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.754711 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" podUID="6768f9ea-68d1-4f33-8e92-fa224878ee1c" containerName="payload-pre-processing" containerID="cri-o://5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8" gracePeriod=30 Nov 01 16:30:34.792583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.792534 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:34.891982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:34.891944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:35.023807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.013011 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:35.186601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.186569 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:35.261666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.261637 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:30:35.265103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.265081 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:30:35.268102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.268073 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:35.387631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.387598 4573 generic.go:356] "Generic (PLEG): container finished" podID="719e9e9e-90ae-4178-b113-96166fdeb425" containerID="9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5" exitCode=0 Nov 01 16:30:35.387834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.387649 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" Nov 01 16:30:35.387834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.387678 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" event={"ID":"719e9e9e-90ae-4178-b113-96166fdeb425","Type":"ContainerDied","Data":"9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5"} Nov 01 16:30:35.387834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.387731 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv" event={"ID":"719e9e9e-90ae-4178-b113-96166fdeb425","Type":"ContainerDied","Data":"6d799361a162ff9aae326e95290f925022c5a1fa7752841c53ef1c0e9463c448"} Nov 01 16:30:35.387834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.387772 4573 scope.go:122] "RemoveContainer" containerID="9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5" Nov 01 16:30:35.389167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.389143 4573 generic.go:356] "Generic (PLEG): container finished" podID="6768f9ea-68d1-4f33-8e92-fa224878ee1c" containerID="5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8" exitCode=0 Nov 01 16:30:35.389261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.389187 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" Nov 01 16:30:35.389261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.389221 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" event={"ID":"6768f9ea-68d1-4f33-8e92-fa224878ee1c","Type":"ContainerDied","Data":"5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8"} Nov 01 16:30:35.389355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.389276 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm" event={"ID":"6768f9ea-68d1-4f33-8e92-fa224878ee1c","Type":"ContainerDied","Data":"627dded5836229bf59c1a23210d5f97286f26797fd9bab7d43884127084d69c7"} Nov 01 16:30:35.389412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.389400 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" podUID="df545125-bbe8-4ed1-8d81-012278b7eeaa" containerName="payload-processing" containerID="cri-o://d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b" gracePeriod=30 Nov 01 16:30:35.398896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.398878 4573 scope.go:122] "RemoveContainer" containerID="9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5" Nov 01 16:30:35.399163 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:35.399141 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5\": container with ID starting with 9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5 not found: ID does not exist" containerID="9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5" Nov 01 16:30:35.399224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.399170 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5"} err="failed to get container status \"9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5\": rpc error: code = NotFound desc = could not find container \"9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5\": container with ID starting with 9a85ee9b83bb7b44c507d463cfab62fb5c56038aef1d286b02d88786750f48d5 not found: ID does not exist" Nov 01 16:30:35.399224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.399187 4573 scope.go:122] "RemoveContainer" containerID="5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8" Nov 01 16:30:35.408784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.408731 4573 scope.go:122] "RemoveContainer" containerID="5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8" Nov 01 16:30:35.409073 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:35.409055 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8\": container with ID starting with 5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8 not found: ID does not exist" containerID="5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8" Nov 01 16:30:35.409152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.409081 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8"} err="failed to get container status \"5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8\": rpc error: code = NotFound desc = could not find container \"5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8\": container with ID starting with 5b8b7fb080a3ecb0967363a7ec1c50d133bf729d045c0b53b130414dead763c8 not found: ID does not exist" Nov 01 16:30:35.419731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.419701 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6768f9ea-68d1-4f33-8e92-fa224878ee1c-kube-api-access-5kc89\" (UniqueName: \"kubernetes.io/projected/6768f9ea-68d1-4f33-8e92-fa224878ee1c-kube-api-access-5kc89\") pod \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\" (UID: \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\") " Nov 01 16:30:35.419905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.419831 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6768f9ea-68d1-4f33-8e92-fa224878ee1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768f9ea-68d1-4f33-8e92-fa224878ee1c-plugins-config-volume\") pod \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\" (UID: \"6768f9ea-68d1-4f33-8e92-fa224878ee1c\") " Nov 01 16:30:35.419905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.419882 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719e9e9e-90ae-4178-b113-96166fdeb425-kube-api-access-kpftt\" (UniqueName: \"kubernetes.io/projected/719e9e9e-90ae-4178-b113-96166fdeb425-kube-api-access-kpftt\") pod \"719e9e9e-90ae-4178-b113-96166fdeb425\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " Nov 01 16:30:35.421453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.419954 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719e9e9e-90ae-4178-b113-96166fdeb425-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719e9e9e-90ae-4178-b113-96166fdeb425-openshift-service-ca\") pod \"719e9e9e-90ae-4178-b113-96166fdeb425\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " Nov 01 16:30:35.421453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.420003 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-metrics-certs\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-metrics-certs\") pod \"719e9e9e-90ae-4178-b113-96166fdeb425\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " Nov 01 16:30:35.421453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.420098 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-maas-api-tls\") pod \"719e9e9e-90ae-4178-b113-96166fdeb425\" (UID: \"719e9e9e-90ae-4178-b113-96166fdeb425\") " Nov 01 16:30:35.421453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.420527 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6768f9ea-68d1-4f33-8e92-fa224878ee1c-plugins-config-volume" pod "6768f9ea-68d1-4f33-8e92-fa224878ee1c" (UID: "6768f9ea-68d1-4f33-8e92-fa224878ee1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:35.421453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.420815 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719e9e9e-90ae-4178-b113-96166fdeb425-openshift-service-ca" pod "719e9e9e-90ae-4178-b113-96166fdeb425" (UID: "719e9e9e-90ae-4178-b113-96166fdeb425"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:35.422568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.422514 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/719e9e9e-90ae-4178-b113-96166fdeb425-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:35.422568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.422541 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6768f9ea-68d1-4f33-8e92-fa224878ee1c-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:35.423026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.422995 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-maas-api-tls" pod "719e9e9e-90ae-4178-b113-96166fdeb425" (UID: "719e9e9e-90ae-4178-b113-96166fdeb425"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:35.424625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.424601 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-metrics-certs" pod "719e9e9e-90ae-4178-b113-96166fdeb425" (UID: "719e9e9e-90ae-4178-b113-96166fdeb425"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:35.425134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.425104 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6768f9ea-68d1-4f33-8e92-fa224878ee1c-kube-api-access-5kc89" pod "6768f9ea-68d1-4f33-8e92-fa224878ee1c" (UID: "6768f9ea-68d1-4f33-8e92-fa224878ee1c"). InnerVolumeSpecName "kube-api-access-5kc89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:35.425320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.425304 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719e9e9e-90ae-4178-b113-96166fdeb425-kube-api-access-kpftt" pod "719e9e9e-90ae-4178-b113-96166fdeb425" (UID: "719e9e9e-90ae-4178-b113-96166fdeb425"). InnerVolumeSpecName "kube-api-access-kpftt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:35.524400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.524323 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kc89\" (UniqueName: \"kubernetes.io/projected/6768f9ea-68d1-4f33-8e92-fa224878ee1c-kube-api-access-5kc89\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:35.524400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.524350 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpftt\" (UniqueName: \"kubernetes.io/projected/719e9e9e-90ae-4178-b113-96166fdeb425-kube-api-access-kpftt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:35.524400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.524360 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:35.524400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.524369 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/719e9e9e-90ae-4178-b113-96166fdeb425-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:35.725557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.725524 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm"] Nov 01 16:30:35.728654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.728537 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:35.732332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.732309 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e1e630-5b5f67bc79-j5ltm"] Nov 01 16:30:35.753442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.753414 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv"] Nov 01 16:30:35.761455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:35.761421 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e1e630-bb564856b-ltzjv"] Nov 01 16:30:36.290780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:36.290702 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:36.460921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:36.460892 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6768f9ea-68d1-4f33-8e92-fa224878ee1c" path="/var/lib/kubelet/pods/6768f9ea-68d1-4f33-8e92-fa224878ee1c/volumes" Nov 01 16:30:36.461385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:36.461357 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719e9e9e-90ae-4178-b113-96166fdeb425" path="/var/lib/kubelet/pods/719e9e9e-90ae-4178-b113-96166fdeb425/volumes" Nov 01 16:30:36.942623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:36.942590 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:37.033587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.033547 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:37.040394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.040368 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq"] Nov 01 16:30:37.041081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041066 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7e74a675-d484-496a-a223-1d03ca77f424" containerName="payload-pre-processing" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041082 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e74a675-d484-496a-a223-1d03ca77f424" containerName="payload-pre-processing" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041096 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041101 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041110 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041116 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041122 4573 cpu_manager.go:414] "containerMap: removing container" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041126 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041138 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041142 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041150 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041155 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041162 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041166 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041174 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:37.041174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041180 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041186 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041190 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041200 4573 cpu_manager.go:414] "containerMap: removing container" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041206 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041219 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041223 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041231 4573 cpu_manager.go:414] "containerMap: removing container" podUID="839d2919-d17c-4e19-bdba-dcbb222ba1d1" containerName="payload-processing" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041235 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839d2919-d17c-4e19-bdba-dcbb222ba1d1" containerName="payload-processing" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041243 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041247 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041253 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041257 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041268 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041274 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041288 4573 cpu_manager.go:414] "containerMap: removing container" podUID="719e9e9e-90ae-4178-b113-96166fdeb425" containerName="maas-api" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041295 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719e9e9e-90ae-4178-b113-96166fdeb425" containerName="maas-api" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041303 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041308 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041316 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041320 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041326 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6768f9ea-68d1-4f33-8e92-fa224878ee1c" containerName="payload-pre-processing" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041331 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6768f9ea-68d1-4f33-8e92-fa224878ee1c" containerName="payload-pre-processing" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041344 4573 cpu_manager.go:414] "containerMap: removing container" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041349 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041362 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041369 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041377 4573 cpu_manager.go:414] "containerMap: removing container" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041381 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041482 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6768f9ea-68d1-4f33-8e92-fa224878ee1c" containerName="payload-pre-processing" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041490 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041498 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041504 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041513 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041521 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041527 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041534 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041540 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e74a675-d484-496a-a223-1d03ca77f424" containerName="payload-pre-processing" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041547 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041552 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041557 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041562 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041566 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041571 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041577 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041583 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eaa91b1-d9bf-4ab2-8b66-1a3588c94ee6" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041591 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041599 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="839d2919-d17c-4e19-bdba-dcbb222ba1d1" containerName="payload-processing" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041606 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041613 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041619 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="719e9e9e-90ae-4178-b113-96166fdeb425" containerName="maas-api" Nov 01 16:30:37.041650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.041624 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:37.046947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.046918 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:37.053464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.053440 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq"] Nov 01 16:30:37.054180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.054158 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-028c46\"" Nov 01 16:30:37.154988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.154845 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba74750-f945-4e87-8d55-221be97e33a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq\" (UID: \"eba74750-f945-4e87-8d55-221be97e33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:37.155191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.155159 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f64h\" (UniqueName: \"kubernetes.io/projected/eba74750-f945-4e87-8d55-221be97e33a4-kube-api-access-5f64h\") pod \"payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq\" (UID: \"eba74750-f945-4e87-8d55-221be97e33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:37.258168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.246935 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" podUID="df545125-bbe8-4ed1-8d81-012278b7eeaa" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.248:9004: connect: connection refused" Nov 01 16:30:37.267772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.264777 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f64h\" (UniqueName: \"kubernetes.io/projected/eba74750-f945-4e87-8d55-221be97e33a4-kube-api-access-5f64h\") pod \"payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq\" (UID: \"eba74750-f945-4e87-8d55-221be97e33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:37.267772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.264912 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba74750-f945-4e87-8d55-221be97e33a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq\" (UID: \"eba74750-f945-4e87-8d55-221be97e33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:37.267772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.265951 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba74750-f945-4e87-8d55-221be97e33a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq\" (UID: \"eba74750-f945-4e87-8d55-221be97e33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:37.281772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.274478 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:37.294579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.294531 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f64h\" (UniqueName: \"kubernetes.io/projected/eba74750-f945-4e87-8d55-221be97e33a4-kube-api-access-5f64h\") pod \"payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq\" (UID: \"eba74750-f945-4e87-8d55-221be97e33a4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:37.365044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.365014 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:37.365549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.365510 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:37.550397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.550314 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:37.652663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.652630 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:37.656395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.656365 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq"] Nov 01 16:30:37.656586 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:37.656565 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba74750_f945_4e87_8d55_221be97e33a4.slice/crio-28a42c5ebf9d4c0c72057ac41282b0b07ba2359e091a2fdc25b8324df2b36499 WatchSource:0}: Error finding container 28a42c5ebf9d4c0c72057ac41282b0b07ba2359e091a2fdc25b8324df2b36499: Status 404 returned error can't find the container with id 28a42c5ebf9d4c0c72057ac41282b0b07ba2359e091a2fdc25b8324df2b36499 Nov 01 16:30:37.997699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:37.997668 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:38.023567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.023538 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p"] Nov 01 16:30:38.027551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.027529 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1754432c-7w57p"] Nov 01 16:30:38.190745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.190708 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8"] Nov 01 16:30:38.191650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191626 4573 cpu_manager.go:414] "containerMap: removing container" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:38.191650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191647 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191668 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191677 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1343be-17df-4d00-a24b-37426a338c89" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191695 4573 cpu_manager.go:414] "containerMap: removing container" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191702 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191726 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191733 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191765 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191772 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c48c30a-0215-4276-95a8-f1ebced71602" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191802 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:38.191882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191809 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:38.192244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191971 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="693232a8-19dd-44b9-ab5e-b073b3dbd82c" containerName="cleanup" Nov 01 16:30:38.192244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.191988 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7812173e-6df9-4f18-a704-b316ca57c568" containerName="cleanup" Nov 01 16:30:38.192244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.192001 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0a40d1-ad36-416e-94d5-3ca525262685" containerName="cleanup" Nov 01 16:30:38.192244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.192011 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8ae3328-ea4c-43ff-87a7-2a6b6e9a347d" containerName="cleanup" Nov 01 16:30:38.192244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.192025 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f3372f-7731-4b2d-842c-3be1a6c71855" containerName="cleanup" Nov 01 16:30:38.195702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.195684 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.207771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.204483 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8"] Nov 01 16:30:38.264619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.264591 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:30:38.264619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.264614 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:30:38.281504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.281474 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhj2v\" (UniqueName: \"kubernetes.io/projected/308c24e1-72df-4537-8453-24031a35c955-kube-api-access-zhj2v\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.281669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.281637 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c24e1-72df-4537-8453-24031a35c955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.281826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.281807 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c24e1-72df-4537-8453-24031a35c955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.383083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.383053 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c24e1-72df-4537-8453-24031a35c955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.383277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.383161 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhj2v\" (UniqueName: \"kubernetes.io/projected/308c24e1-72df-4537-8453-24031a35c955-kube-api-access-zhj2v\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.383277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.383243 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c24e1-72df-4537-8453-24031a35c955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.383926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.383903 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c24e1-72df-4537-8453-24031a35c955-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.385595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.385571 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c24e1-72df-4537-8453-24031a35c955-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.391008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.390984 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhj2v\" (UniqueName: \"kubernetes.io/projected/308c24e1-72df-4537-8453-24031a35c955-kube-api-access-zhj2v\") pod \"maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.428650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.428622 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" event={"ID":"eba74750-f945-4e87-8d55-221be97e33a4","Type":"ContainerStarted","Data":"aceb1e7f814946480be1b6919c80b1a58c92e877e23003d6081a3690ae4b133b"} Nov 01 16:30:38.428827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.428667 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" event={"ID":"eba74750-f945-4e87-8d55-221be97e33a4","Type":"ContainerStarted","Data":"28a42c5ebf9d4c0c72057ac41282b0b07ba2359e091a2fdc25b8324df2b36499"} Nov 01 16:30:38.428827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.428770 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:38.448434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.448360 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" podStartSLOduration=1.448348036 podStartE2EDuration="1.448348036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:38.443396002 +0000 UTC m=+2576.146561938" watchObservedRunningTime="2026-11-01 16:30:38.448348036 +0000 UTC m=+2576.151513971" Nov 01 16:30:38.459969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.459946 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b143285-fc25-4acb-aecc-22c0fe812f3c" path="/var/lib/kubelet/pods/5b143285-fc25-4acb-aecc-22c0fe812f3c/volumes" Nov 01 16:30:38.522964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.522937 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:38.719900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.719806 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:38.845062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:38.845032 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8"] Nov 01 16:30:38.845199 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:38.845035 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308c24e1_72df_4537_8453_24031a35c955.slice/crio-70e597fdf9935a6ff2f74ed60d4c82b4117c4ed0a0c7b25c477b73428e5b1d60 WatchSource:0}: Error finding container 70e597fdf9935a6ff2f74ed60d4c82b4117c4ed0a0c7b25c477b73428e5b1d60: Status 404 returned error can't find the container with id 70e597fdf9935a6ff2f74ed60d4c82b4117c4ed0a0c7b25c477b73428e5b1d60 Nov 01 16:30:39.438159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:39.438127 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" event={"ID":"308c24e1-72df-4537-8453-24031a35c955","Type":"ContainerStarted","Data":"9508774ca51006fa64b144e59de6936e5fccaf2b92775bce3ef9bf39724fd895"} Nov 01 16:30:39.438594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:39.438190 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" event={"ID":"308c24e1-72df-4537-8453-24031a35c955","Type":"ContainerStarted","Data":"70e597fdf9935a6ff2f74ed60d4c82b4117c4ed0a0c7b25c477b73428e5b1d60"} Nov 01 16:30:39.456487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:39.456442 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" podStartSLOduration=1.456428652 podStartE2EDuration="1.456428652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:39.454407472 +0000 UTC m=+2577.157573408" watchObservedRunningTime="2026-11-01 16:30:39.456428652 +0000 UTC m=+2577.159594607" Nov 01 16:30:39.652438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:39.652405 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:40.446666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:40.446628 4573 generic.go:356] "Generic (PLEG): container finished" podID="308c24e1-72df-4537-8453-24031a35c955" containerID="9508774ca51006fa64b144e59de6936e5fccaf2b92775bce3ef9bf39724fd895" exitCode=0 Nov 01 16:30:40.447126 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:40.446683 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" event={"ID":"308c24e1-72df-4537-8453-24031a35c955","Type":"ContainerDied","Data":"9508774ca51006fa64b144e59de6936e5fccaf2b92775bce3ef9bf39724fd895"} Nov 01 16:30:40.472065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:40.472034 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:40.663666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:40.663635 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:41.233130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.233098 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:41.641135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.641109 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:41.665131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.665105 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308c24e1-72df-4537-8453-24031a35c955-kube-api-access-zhj2v\" (UniqueName: \"kubernetes.io/projected/308c24e1-72df-4537-8453-24031a35c955-kube-api-access-zhj2v\") pod \"308c24e1-72df-4537-8453-24031a35c955\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " Nov 01 16:30:41.665282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.665170 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/308c24e1-72df-4537-8453-24031a35c955-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c24e1-72df-4537-8453-24031a35c955-maas-api-xks-ca\") pod \"308c24e1-72df-4537-8453-24031a35c955\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " Nov 01 16:30:41.665282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.665202 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308c24e1-72df-4537-8453-24031a35c955-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c24e1-72df-4537-8453-24031a35c955-maas-api-openshift-ca\") pod \"308c24e1-72df-4537-8453-24031a35c955\" (UID: \"308c24e1-72df-4537-8453-24031a35c955\") " Nov 01 16:30:41.666299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.665845 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308c24e1-72df-4537-8453-24031a35c955-maas-api-openshift-ca" pod "308c24e1-72df-4537-8453-24031a35c955" (UID: "308c24e1-72df-4537-8453-24031a35c955"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:41.669907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.669875 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308c24e1-72df-4537-8453-24031a35c955-kube-api-access-zhj2v" pod "308c24e1-72df-4537-8453-24031a35c955" (UID: "308c24e1-72df-4537-8453-24031a35c955"). InnerVolumeSpecName "kube-api-access-zhj2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:41.670786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.670731 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/308c24e1-72df-4537-8453-24031a35c955-maas-api-xks-ca" pod "308c24e1-72df-4537-8453-24031a35c955" (UID: "308c24e1-72df-4537-8453-24031a35c955"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:41.699028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.699005 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:30:41.766449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.766354 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df545125-bbe8-4ed1-8d81-012278b7eeaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df545125-bbe8-4ed1-8d81-012278b7eeaa-plugins-config-volume\") pod \"df545125-bbe8-4ed1-8d81-012278b7eeaa\" (UID: \"df545125-bbe8-4ed1-8d81-012278b7eeaa\") " Nov 01 16:30:41.766621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.766470 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df545125-bbe8-4ed1-8d81-012278b7eeaa-kube-api-access-bjcf2\" (UniqueName: \"kubernetes.io/projected/df545125-bbe8-4ed1-8d81-012278b7eeaa-kube-api-access-bjcf2\") pod \"df545125-bbe8-4ed1-8d81-012278b7eeaa\" (UID: \"df545125-bbe8-4ed1-8d81-012278b7eeaa\") " Nov 01 16:30:41.766864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.766838 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df545125-bbe8-4ed1-8d81-012278b7eeaa-plugins-config-volume" pod "df545125-bbe8-4ed1-8d81-012278b7eeaa" (UID: "df545125-bbe8-4ed1-8d81-012278b7eeaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:41.766977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.766931 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhj2v\" (UniqueName: \"kubernetes.io/projected/308c24e1-72df-4537-8453-24031a35c955-kube-api-access-zhj2v\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:41.766977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.766951 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/308c24e1-72df-4537-8453-24031a35c955-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:41.766977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.766965 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/308c24e1-72df-4537-8453-24031a35c955-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:41.768874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.768850 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df545125-bbe8-4ed1-8d81-012278b7eeaa-kube-api-access-bjcf2" pod "df545125-bbe8-4ed1-8d81-012278b7eeaa" (UID: "df545125-bbe8-4ed1-8d81-012278b7eeaa"). InnerVolumeSpecName "kube-api-access-bjcf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:41.868774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.868579 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df545125-bbe8-4ed1-8d81-012278b7eeaa-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:41.868774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:41.868618 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjcf2\" (UniqueName: \"kubernetes.io/projected/df545125-bbe8-4ed1-8d81-012278b7eeaa-kube-api-access-bjcf2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:42.048696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.048617 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:42.464588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.464555 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" Nov 01 16:30:42.464812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.464608 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8" event={"ID":"308c24e1-72df-4537-8453-24031a35c955","Type":"ContainerDied","Data":"70e597fdf9935a6ff2f74ed60d4c82b4117c4ed0a0c7b25c477b73428e5b1d60"} Nov 01 16:30:42.464812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.464662 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70e597fdf9935a6ff2f74ed60d4c82b4117c4ed0a0c7b25c477b73428e5b1d60" Nov 01 16:30:42.466471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.466444 4573 generic.go:356] "Generic (PLEG): container finished" podID="df545125-bbe8-4ed1-8d81-012278b7eeaa" containerID="d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b" exitCode=0 Nov 01 16:30:42.466623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.466486 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" Nov 01 16:30:42.466623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.466529 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" event={"ID":"df545125-bbe8-4ed1-8d81-012278b7eeaa","Type":"ContainerDied","Data":"d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b"} Nov 01 16:30:42.466623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.466590 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm" event={"ID":"df545125-bbe8-4ed1-8d81-012278b7eeaa","Type":"ContainerDied","Data":"7e012681f57121f1c3b6584be866ac4d49c1fb253aa4e85d171f78f1cfc654d2"} Nov 01 16:30:42.466623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.466620 4573 scope.go:122] "RemoveContainer" containerID="d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b" Nov 01 16:30:42.487645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.487618 4573 scope.go:122] "RemoveContainer" containerID="d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b" Nov 01 16:30:42.488179 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:42.488039 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b\": container with ID starting with d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b not found: ID does not exist" containerID="d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b" Nov 01 16:30:42.488179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.488074 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b"} err="failed to get container status \"d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b\": rpc error: code = NotFound desc = could not find container \"d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b\": container with ID starting with d1f442647e3c2c2e1f82d5b0275b4cfb7f3de596a2e4b6172e6187b7efe2439b not found: ID does not exist" Nov 01 16:30:42.498325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.498301 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm"] Nov 01 16:30:42.506541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.506514 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e1e630-557c686fcd-w6sbm"] Nov 01 16:30:42.966475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.966442 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:42.980154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.980125 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87"] Nov 01 16:30:42.994487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:42.994450 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g"] Nov 01 16:30:43.071031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:43.071002 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:43.765192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:43.765160 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:44.327860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.327827 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r"] Nov 01 16:30:44.328881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.328854 4573 cpu_manager.go:414] "containerMap: removing container" podUID="df545125-bbe8-4ed1-8d81-012278b7eeaa" containerName="payload-processing" Nov 01 16:30:44.328881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.328880 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df545125-bbe8-4ed1-8d81-012278b7eeaa" containerName="payload-processing" Nov 01 16:30:44.329111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.328902 4573 cpu_manager.go:414] "containerMap: removing container" podUID="308c24e1-72df-4537-8453-24031a35c955" containerName="revoke-keys" Nov 01 16:30:44.329111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.328910 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308c24e1-72df-4537-8453-24031a35c955" containerName="revoke-keys" Nov 01 16:30:44.329214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.329110 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="df545125-bbe8-4ed1-8d81-012278b7eeaa" containerName="payload-processing" Nov 01 16:30:44.329214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.329129 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="308c24e1-72df-4537-8453-24031a35c955" containerName="revoke-keys" Nov 01 16:30:44.362286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.335497 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.371404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.371375 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r"] Nov 01 16:30:44.401343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401310 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44daf16c-1043-4b4e-bb2d-27aef63adf38-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.401505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401452 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.401505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401498 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.401651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401632 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj87c\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-kube-api-access-lj87c\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.401799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401743 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.401902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401833 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.401902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401888 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.402003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401951 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.402003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.401997 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.425677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.425657 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-028c46-openshift-default-dockercfg-8z222\"" Nov 01 16:30:44.459287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.459260 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df545125-bbe8-4ed1-8d81-012278b7eeaa" path="/var/lib/kubelet/pods/df545125-bbe8-4ed1-8d81-012278b7eeaa/volumes" Nov 01 16:30:44.503280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503252 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.503420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503291 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.503420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503409 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.503559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503440 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.503622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503580 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44daf16c-1043-4b4e-bb2d-27aef63adf38-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.503683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503635 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-data\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.503789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503685 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-credential-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.503867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503794 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.503867 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503842 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.504004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.503978 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj87c\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-kube-api-access-lj87c\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.504093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.504075 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.504187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.504165 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-certs\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.504433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.504400 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44daf16c-1043-4b4e-bb2d-27aef63adf38-istiod-ca-cert\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.504524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.504404 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-socket\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.505903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.505882 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-envoy\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.506047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.506030 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-podinfo\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.513036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.513016 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj87c\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-kube-api-access-lj87c\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.513143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.513075 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-token\") pod \"e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.554080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.554054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:44.680264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.680236 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:44.968047 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:44.968017 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44daf16c_1043_4b4e_bb2d_27aef63adf38.slice/crio-ffbc5b081804e15d4567135c0b168d8ab8f51da3bb862dd7467eb662b17ef130 WatchSource:0}: Error finding container ffbc5b081804e15d4567135c0b168d8ab8f51da3bb862dd7467eb662b17ef130: Status 404 returned error can't find the container with id ffbc5b081804e15d4567135c0b168d8ab8f51da3bb862dd7467eb662b17ef130 Nov 01 16:30:44.968249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.968225 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r"] Nov 01 16:30:44.971447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.971407 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:44.971561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.971494 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:44.971561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:44.971538 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:30:45.199045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.199012 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:45.349428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.349333 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x"] Nov 01 16:30:45.463594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.463391 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:45.494980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.494943 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" event={"ID":"44daf16c-1043-4b4e-bb2d-27aef63adf38","Type":"ContainerStarted","Data":"c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b"} Nov 01 16:30:45.495189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.495009 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" event={"ID":"44daf16c-1043-4b4e-bb2d-27aef63adf38","Type":"ContainerStarted","Data":"ffbc5b081804e15d4567135c0b168d8ab8f51da3bb862dd7467eb662b17ef130"} Nov 01 16:30:45.522787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.521849 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" podStartSLOduration=1.521829067 podStartE2EDuration="1.521829067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:45.517617581 +0000 UTC m=+2583.220783520" watchObservedRunningTime="2026-11-01 16:30:45.521829067 +0000 UTC m=+2583.224995006" Nov 01 16:30:45.560398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.560344 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:45.678875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.678632 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:45.691697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:45.691634 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:46.184775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:46.184647 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:46.688895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:46.688844 4573 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 16:30:46.689323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:46.688906 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 16:30:46.986832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:46.984989 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:47.652672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:47.652639 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:47.685775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:47.685389 4573 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 16:30:47.685775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:47.685452 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 16:30:48.290817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:48.290780 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm"] Nov 01 16:30:48.342902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:48.342862 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:48.439498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:48.439465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:48.638776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:48.638414 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:48.700779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:48.700077 4573 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 16:30:48.700779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:48.700171 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 16:30:49.092383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:49.092303 4573 scope.go:122] "RemoveContainer" containerID="4405967305bd31d106beacca517c1322f7730e456cb6af20b18b27576f827b39" Nov 01 16:30:49.414659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:49.414629 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:49.449728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:49.449686 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:30:49.684707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:49.684186 4573 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 16:30:49.684707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:49.684265 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 16:30:50.350419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.350383 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" podUID="09f12a70-d45e-4340-bdaf-c61d3af3d130" containerName="payload-processing" containerID="cri-o://731367e3ff00edc8d8de0aa2d006306f5e5d345c67fbfd43f085d64d00a3745a" gracePeriod=25 Nov 01 16:30:50.534844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.534810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:50.692546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.690465 4573 patch_prober.go:29] interesting pod/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body= Nov 01 16:30:50.692546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.690527 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" Nov 01 16:30:50.702489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.702239 4573 generic.go:356] "Generic (PLEG): container finished" podID="09f12a70-d45e-4340-bdaf-c61d3af3d130" containerID="731367e3ff00edc8d8de0aa2d006306f5e5d345c67fbfd43f085d64d00a3745a" exitCode=0 Nov 01 16:30:50.702489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.702422 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" event={"ID":"09f12a70-d45e-4340-bdaf-c61d3af3d130","Type":"ContainerDied","Data":"731367e3ff00edc8d8de0aa2d006306f5e5d345c67fbfd43f085d64d00a3745a"} Nov 01 16:30:50.747056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.747030 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:50.887343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.887302 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f12a70-d45e-4340-bdaf-c61d3af3d130-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f12a70-d45e-4340-bdaf-c61d3af3d130-plugins-config-volume\") pod \"09f12a70-d45e-4340-bdaf-c61d3af3d130\" (UID: \"09f12a70-d45e-4340-bdaf-c61d3af3d130\") " Nov 01 16:30:50.887549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.887399 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f12a70-d45e-4340-bdaf-c61d3af3d130-kube-api-access-qq8wt\" (UniqueName: \"kubernetes.io/projected/09f12a70-d45e-4340-bdaf-c61d3af3d130-kube-api-access-qq8wt\") pod \"09f12a70-d45e-4340-bdaf-c61d3af3d130\" (UID: \"09f12a70-d45e-4340-bdaf-c61d3af3d130\") " Nov 01 16:30:50.887730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.887705 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f12a70-d45e-4340-bdaf-c61d3af3d130-plugins-config-volume" pod "09f12a70-d45e-4340-bdaf-c61d3af3d130" (UID: "09f12a70-d45e-4340-bdaf-c61d3af3d130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:50.887917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.887901 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09f12a70-d45e-4340-bdaf-c61d3af3d130-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:50.889698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.889673 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f12a70-d45e-4340-bdaf-c61d3af3d130-kube-api-access-qq8wt" pod "09f12a70-d45e-4340-bdaf-c61d3af3d130" (UID: "09f12a70-d45e-4340-bdaf-c61d3af3d130"). InnerVolumeSpecName "kube-api-access-qq8wt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:50.989593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:50.989509 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq8wt\" (UniqueName: \"kubernetes.io/projected/09f12a70-d45e-4340-bdaf-c61d3af3d130-kube-api-access-qq8wt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:51.387740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.387706 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:51.685331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.685239 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:51.685852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.685827 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:51.686766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.686706 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:30:51.717136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.717103 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" Nov 01 16:30:51.717136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.717129 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x" event={"ID":"09f12a70-d45e-4340-bdaf-c61d3af3d130","Type":"ContainerDied","Data":"6898ecaa60dc086209a460636cd913e63883c4c7d3f9627cc1fc4369ea087e32"} Nov 01 16:30:51.718124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.717220 4573 scope.go:122] "RemoveContainer" containerID="731367e3ff00edc8d8de0aa2d006306f5e5d345c67fbfd43f085d64d00a3745a" Nov 01 16:30:51.794965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.794923 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x"] Nov 01 16:30:51.800885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.800827 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-7f499968b7-xdg7x"] Nov 01 16:30:51.975694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.975615 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj"] Nov 01 16:30:51.976338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.976323 4573 cpu_manager.go:414] "containerMap: removing container" podUID="09f12a70-d45e-4340-bdaf-c61d3af3d130" containerName="payload-processing" Nov 01 16:30:51.976386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.976339 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f12a70-d45e-4340-bdaf-c61d3af3d130" containerName="payload-processing" Nov 01 16:30:51.976475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.976466 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f12a70-d45e-4340-bdaf-c61d3af3d130" containerName="payload-processing" Nov 01 16:30:51.981731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.981702 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:51.985512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.985484 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:30:51.985703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.985511 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:30:51.995379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:51.991126 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj"] Nov 01 16:30:52.004031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.004002 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.004167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.004108 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lrj\" (UniqueName: \"kubernetes.io/projected/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-kube-api-access-b5lrj\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.004279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.004261 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.105494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.105462 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lrj\" (UniqueName: \"kubernetes.io/projected/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-kube-api-access-b5lrj\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.105685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.105608 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.105808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.105733 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.106358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.106327 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.107971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.107948 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.113571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.113553 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lrj\" (UniqueName: \"kubernetes.io/projected/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-kube-api-access-b5lrj\") pod \"maas-api-revoke-sub-14b4bf21-z7ldj\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.252722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.252624 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:52.305522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.305359 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:52.464790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.464733 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f12a70-d45e-4340-bdaf-c61d3af3d130" path="/var/lib/kubelet/pods/09f12a70-d45e-4340-bdaf-c61d3af3d130/volumes" Nov 01 16:30:52.622110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.621366 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj"] Nov 01 16:30:52.623210 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:52.623182 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cd89f5_e5e6_4f6b_968e_dccc8d393e53.slice/crio-ca2055443a5d9579d16b1249757196e6792dc917fd6ab09dbe48f8896723d2e5 WatchSource:0}: Error finding container ca2055443a5d9579d16b1249757196e6792dc917fd6ab09dbe48f8896723d2e5: Status 404 returned error can't find the container with id ca2055443a5d9579d16b1249757196e6792dc917fd6ab09dbe48f8896723d2e5 Nov 01 16:30:52.726293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:52.726259 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" event={"ID":"20cd89f5-e5e6-4f6b-968e-dccc8d393e53","Type":"ContainerStarted","Data":"ca2055443a5d9579d16b1249757196e6792dc917fd6ab09dbe48f8896723d2e5"} Nov 01 16:30:53.137667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.137636 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:53.291591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.291491 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" podUID="5ae37fe0-f2ca-4b70-9ac8-27843820b864" containerName="payload-pre-processing" containerID="cri-o://3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211" gracePeriod=25 Nov 01 16:30:53.572280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.572250 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 16:30:53.644112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.644084 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae37fe0-f2ca-4b70-9ac8-27843820b864-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae37fe0-f2ca-4b70-9ac8-27843820b864-plugins-config-volume\") pod \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\" (UID: \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\") " Nov 01 16:30:53.644257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.644217 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae37fe0-f2ca-4b70-9ac8-27843820b864-kube-api-access-5zvsc\" (UniqueName: \"kubernetes.io/projected/5ae37fe0-f2ca-4b70-9ac8-27843820b864-kube-api-access-5zvsc\") pod \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\" (UID: \"5ae37fe0-f2ca-4b70-9ac8-27843820b864\") " Nov 01 16:30:53.644504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.644479 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae37fe0-f2ca-4b70-9ac8-27843820b864-plugins-config-volume" pod "5ae37fe0-f2ca-4b70-9ac8-27843820b864" (UID: "5ae37fe0-f2ca-4b70-9ac8-27843820b864"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:53.644721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.644706 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ae37fe0-f2ca-4b70-9ac8-27843820b864-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:53.646377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.646359 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae37fe0-f2ca-4b70-9ac8-27843820b864-kube-api-access-5zvsc" pod "5ae37fe0-f2ca-4b70-9ac8-27843820b864" (UID: "5ae37fe0-f2ca-4b70-9ac8-27843820b864"). InnerVolumeSpecName "kube-api-access-5zvsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:53.736932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.736896 4573 generic.go:356] "Generic (PLEG): container finished" podID="20cd89f5-e5e6-4f6b-968e-dccc8d393e53" containerID="347d4bf6785d2d7f8341527f5320099cd46d7ae0147506de616b617b3cf4b80e" exitCode=0 Nov 01 16:30:53.737376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.736982 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" event={"ID":"20cd89f5-e5e6-4f6b-968e-dccc8d393e53","Type":"ContainerDied","Data":"347d4bf6785d2d7f8341527f5320099cd46d7ae0147506de616b617b3cf4b80e"} Nov 01 16:30:53.738627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.738602 4573 generic.go:356] "Generic (PLEG): container finished" podID="5ae37fe0-f2ca-4b70-9ac8-27843820b864" containerID="3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211" exitCode=0 Nov 01 16:30:53.738813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.738654 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" Nov 01 16:30:53.738813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.738671 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" event={"ID":"5ae37fe0-f2ca-4b70-9ac8-27843820b864","Type":"ContainerDied","Data":"3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211"} Nov 01 16:30:53.738813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.738710 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm" event={"ID":"5ae37fe0-f2ca-4b70-9ac8-27843820b864","Type":"ContainerDied","Data":"c00b798c26242a351e3d1586d3d504dca7fb46170de825a8e3c9ba070ccebae6"} Nov 01 16:30:53.738813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.738732 4573 scope.go:122] "RemoveContainer" containerID="3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211" Nov 01 16:30:53.746570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.746547 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zvsc\" (UniqueName: \"kubernetes.io/projected/5ae37fe0-f2ca-4b70-9ac8-27843820b864-kube-api-access-5zvsc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:53.750469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.750448 4573 scope.go:122] "RemoveContainer" containerID="3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211" Nov 01 16:30:53.750888 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:53.750861 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211\": container with ID starting with 3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211 not found: ID does not exist" containerID="3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211" Nov 01 16:30:53.750984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.750897 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211"} err="failed to get container status \"3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211\": rpc error: code = NotFound desc = could not find container \"3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211\": container with ID starting with 3dce7baee981ac9d846d3bb501ae4f2a5ad43825a22f2451c3c01134f593f211 not found: ID does not exist" Nov 01 16:30:53.770800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.770107 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm"] Nov 01 16:30:53.786549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.786524 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-4a083d-cdfc4788c-48knm"] Nov 01 16:30:53.810330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:53.810272 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:54.366932 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:54.366894 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:30:54.459364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:54.459283 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae37fe0-f2ca-4b70-9ac8-27843820b864" path="/var/lib/kubelet/pods/5ae37fe0-f2ca-4b70-9ac8-27843820b864/volumes" Nov 01 16:30:54.687445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:54.687401 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:54.931591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:54.931567 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:55.069587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.069488 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-kube-api-access-b5lrj\" (UniqueName: \"kubernetes.io/projected/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-kube-api-access-b5lrj\") pod \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " Nov 01 16:30:55.069587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.069568 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-xks-ca\") pod \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " Nov 01 16:30:55.069860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.069839 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-openshift-ca\") pod \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\" (UID: \"20cd89f5-e5e6-4f6b-968e-dccc8d393e53\") " Nov 01 16:30:55.070227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.070201 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-openshift-ca" pod "20cd89f5-e5e6-4f6b-968e-dccc8d393e53" (UID: "20cd89f5-e5e6-4f6b-968e-dccc8d393e53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:30:55.070385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.070368 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:55.071578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.071555 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-xks-ca" pod "20cd89f5-e5e6-4f6b-968e-dccc8d393e53" (UID: "20cd89f5-e5e6-4f6b-968e-dccc8d393e53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:55.071686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.071669 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-kube-api-access-b5lrj" pod "20cd89f5-e5e6-4f6b-968e-dccc8d393e53" (UID: "20cd89f5-e5e6-4f6b-968e-dccc8d393e53"). InnerVolumeSpecName "kube-api-access-b5lrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:55.172259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.172228 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5lrj\" (UniqueName: \"kubernetes.io/projected/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-kube-api-access-b5lrj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:55.172259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.172255 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20cd89f5-e5e6-4f6b-968e-dccc8d393e53-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:55.349701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.349617 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:55.457179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.457151 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:55.788156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.788096 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" event={"ID":"20cd89f5-e5e6-4f6b-968e-dccc8d393e53","Type":"ContainerDied","Data":"ca2055443a5d9579d16b1249757196e6792dc917fd6ab09dbe48f8896723d2e5"} Nov 01 16:30:55.788156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.788141 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj" Nov 01 16:30:55.788156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.788159 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca2055443a5d9579d16b1249757196e6792dc917fd6ab09dbe48f8896723d2e5" Nov 01 16:30:55.816703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.816651 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:55.958505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:55.958471 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:56.324624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.324586 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:56.805681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.805637 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:56.914517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.914483 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:56.942595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.942565 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv"] Nov 01 16:30:56.943317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.943296 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5ae37fe0-f2ca-4b70-9ac8-27843820b864" containerName="payload-pre-processing" Nov 01 16:30:56.943447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.943317 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae37fe0-f2ca-4b70-9ac8-27843820b864" containerName="payload-pre-processing" Nov 01 16:30:56.943447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.943350 4573 cpu_manager.go:414] "containerMap: removing container" podUID="20cd89f5-e5e6-4f6b-968e-dccc8d393e53" containerName="revoke-keys" Nov 01 16:30:56.943447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.943358 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cd89f5-e5e6-4f6b-968e-dccc8d393e53" containerName="revoke-keys" Nov 01 16:30:56.943613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.943555 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae37fe0-f2ca-4b70-9ac8-27843820b864" containerName="payload-pre-processing" Nov 01 16:30:56.943613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.943571 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cd89f5-e5e6-4f6b-968e-dccc8d393e53" containerName="revoke-keys" Nov 01 16:30:56.947381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.947360 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:56.953393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.953366 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-6b6cfc\"" Nov 01 16:30:56.953529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.953516 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-6b6cfc-dockercfg-fwtx6\"" Nov 01 16:30:56.954136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:56.954113 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv"] Nov 01 16:30:57.103953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.093579 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmg2\" (UniqueName: \"kubernetes.io/projected/803fc24d-d984-49ee-a476-c6dbb72aa92e-kube-api-access-tvmg2\") pod \"payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv\" (UID: \"803fc24d-d984-49ee-a476-c6dbb72aa92e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:57.103953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.094906 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803fc24d-d984-49ee-a476-c6dbb72aa92e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv\" (UID: \"803fc24d-d984-49ee-a476-c6dbb72aa92e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:57.196020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.195990 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803fc24d-d984-49ee-a476-c6dbb72aa92e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv\" (UID: \"803fc24d-d984-49ee-a476-c6dbb72aa92e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:57.196233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.196072 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmg2\" (UniqueName: \"kubernetes.io/projected/803fc24d-d984-49ee-a476-c6dbb72aa92e-kube-api-access-tvmg2\") pod \"payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv\" (UID: \"803fc24d-d984-49ee-a476-c6dbb72aa92e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:57.196742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.196716 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803fc24d-d984-49ee-a476-c6dbb72aa92e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv\" (UID: \"803fc24d-d984-49ee-a476-c6dbb72aa92e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:57.204036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.204013 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmg2\" (UniqueName: \"kubernetes.io/projected/803fc24d-d984-49ee-a476-c6dbb72aa92e-kube-api-access-tvmg2\") pod \"payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv\" (UID: \"803fc24d-d984-49ee-a476-c6dbb72aa92e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:57.267768 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.267730 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:57.453571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.453540 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:57.571553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.571525 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv"] Nov 01 16:30:57.572208 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:30:57.572175 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803fc24d_d984_49ee_a476_c6dbb72aa92e.slice/crio-2cac6a455c44ef1b1f2eaac97876d1d57518d089f61593149f16f67f8227fbf0 WatchSource:0}: Error finding container 2cac6a455c44ef1b1f2eaac97876d1d57518d089f61593149f16f67f8227fbf0: Status 404 returned error can't find the container with id 2cac6a455c44ef1b1f2eaac97876d1d57518d089f61593149f16f67f8227fbf0 Nov 01 16:30:57.805384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.805296 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" event={"ID":"803fc24d-d984-49ee-a476-c6dbb72aa92e","Type":"ContainerStarted","Data":"c2cebf96215f5529808ad9abb7c27507f38a2c3d7df461ab8ccd8809c45aa494"} Nov 01 16:30:57.805384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.805346 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" event={"ID":"803fc24d-d984-49ee-a476-c6dbb72aa92e","Type":"ContainerStarted","Data":"2cac6a455c44ef1b1f2eaac97876d1d57518d089f61593149f16f67f8227fbf0"} Nov 01 16:30:57.805583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.805432 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:30:57.824533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:57.824474 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" podStartSLOduration=1.8244574999999998 podStartE2EDuration="1.8244575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:30:57.819741459 +0000 UTC m=+2595.522907410" watchObservedRunningTime="2026-11-01 16:30:57.8244575 +0000 UTC m=+2595.527623438" Nov 01 16:30:58.010922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.010872 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" containerID="cri-o://f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc" gracePeriod=45 Nov 01 16:30:58.027394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.027364 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" containerID="cri-o://f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0" gracePeriod=45 Nov 01 16:30:58.179968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.179931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:58.408206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.408176 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:30:58.411649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.411623 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:30:58.516107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516067 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-model-cache\") pod \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " Nov 01 16:30:58.516310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516169 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-dshm\") pod \"7062fcad-aa5b-4c09-b743-fbaa104863c6\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " Nov 01 16:30:58.516310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516222 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-tmp-dir\") pod \"7062fcad-aa5b-4c09-b743-fbaa104863c6\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " Nov 01 16:30:58.516310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516256 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7062fcad-aa5b-4c09-b743-fbaa104863c6-kube-api-access-k7m2r\" (UniqueName: \"kubernetes.io/projected/7062fcad-aa5b-4c09-b743-fbaa104863c6-kube-api-access-k7m2r\") pod \"7062fcad-aa5b-4c09-b743-fbaa104863c6\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " Nov 01 16:30:58.516310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516283 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7062fcad-aa5b-4c09-b743-fbaa104863c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/7062fcad-aa5b-4c09-b743-fbaa104863c6-tls-certs\") pod \"7062fcad-aa5b-4c09-b743-fbaa104863c6\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " Nov 01 16:30:58.516310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516304 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tls-certs\") pod \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " Nov 01 16:30:58.516630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516349 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-model-cache" pod "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" (UID: "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.516630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516592 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-tmp-dir" pod "7062fcad-aa5b-4c09-b743-fbaa104863c6" (UID: "7062fcad-aa5b-4c09-b743-fbaa104863c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.516809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516634 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-model-cache\") pod \"7062fcad-aa5b-4c09-b743-fbaa104863c6\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " Nov 01 16:30:58.516809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516695 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-dshm\") pod \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " Nov 01 16:30:58.516955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516866 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-kube-api-access-xp9fv\" (UniqueName: \"kubernetes.io/projected/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-kube-api-access-xp9fv\") pod \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " Nov 01 16:30:58.516955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516887 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-model-cache" pod "7062fcad-aa5b-4c09-b743-fbaa104863c6" (UID: "7062fcad-aa5b-4c09-b743-fbaa104863c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.516955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516935 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tmp-dir\") pod \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " Nov 01 16:30:58.517127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.516966 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-home\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-home\") pod \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\" (UID: \"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e\") " Nov 01 16:30:58.517127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.517006 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-home\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-home\") pod \"7062fcad-aa5b-4c09-b743-fbaa104863c6\" (UID: \"7062fcad-aa5b-4c09-b743-fbaa104863c6\") " Nov 01 16:30:58.517290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.517263 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tmp-dir" pod "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" (UID: "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.517725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.517697 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-home" pod "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" (UID: "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.517956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.517938 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-home" pod "7062fcad-aa5b-4c09-b743-fbaa104863c6" (UID: "7062fcad-aa5b-4c09-b743-fbaa104863c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.518210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.518187 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.518302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.518216 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.518302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.518234 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.518302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.518260 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.518302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.518282 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.518549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.518302 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.519692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.519648 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tls-certs" pod "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" (UID: "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:58.520677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.520657 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-dshm" pod "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" (UID: "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.520848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.520805 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7062fcad-aa5b-4c09-b743-fbaa104863c6-tls-certs" pod "7062fcad-aa5b-4c09-b743-fbaa104863c6" (UID: "7062fcad-aa5b-4c09-b743-fbaa104863c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:30:58.520848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.520837 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-kube-api-access-xp9fv" pod "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" (UID: "2a7f0f44-adab-46e3-9232-a1a4f3a30e6e"). InnerVolumeSpecName "kube-api-access-xp9fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:58.520943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.520917 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-dshm" pod "7062fcad-aa5b-4c09-b743-fbaa104863c6" (UID: "7062fcad-aa5b-4c09-b743-fbaa104863c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:30:58.520943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.520921 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7062fcad-aa5b-4c09-b743-fbaa104863c6-kube-api-access-k7m2r" pod "7062fcad-aa5b-4c09-b743-fbaa104863c6" (UID: "7062fcad-aa5b-4c09-b743-fbaa104863c6"). InnerVolumeSpecName "kube-api-access-k7m2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:30:58.621771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.619992 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7062fcad-aa5b-4c09-b743-fbaa104863c6-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.621771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.620036 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7m2r\" (UniqueName: \"kubernetes.io/projected/7062fcad-aa5b-4c09-b743-fbaa104863c6-kube-api-access-k7m2r\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.621771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.620053 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7062fcad-aa5b-4c09-b743-fbaa104863c6-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.621771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.620068 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.621771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.620081 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.621771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.620095 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp9fv\" (UniqueName: \"kubernetes.io/projected/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e-kube-api-access-xp9fv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:30:58.814417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.814383 4573 generic.go:356] "Generic (PLEG): container finished" podID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerID="f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0" exitCode=0 Nov 01 16:30:58.814603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.814431 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" Nov 01 16:30:58.814603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.814470 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" event={"ID":"7062fcad-aa5b-4c09-b743-fbaa104863c6","Type":"ContainerDied","Data":"f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0"} Nov 01 16:30:58.814603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.814511 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g" event={"ID":"7062fcad-aa5b-4c09-b743-fbaa104863c6","Type":"ContainerDied","Data":"6425046e453a71a89d191082d459de0ab32a4e16a1d22be49ce13cf2e40abe40"} Nov 01 16:30:58.814603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.814531 4573 scope.go:122] "RemoveContainer" containerID="f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0" Nov 01 16:30:58.816041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.816020 4573 generic.go:356] "Generic (PLEG): container finished" podID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerID="f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc" exitCode=0 Nov 01 16:30:58.816153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.816065 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" Nov 01 16:30:58.816153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.816116 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" event={"ID":"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e","Type":"ContainerDied","Data":"f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc"} Nov 01 16:30:58.816287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.816153 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87" event={"ID":"2a7f0f44-adab-46e3-9232-a1a4f3a30e6e","Type":"ContainerDied","Data":"b00e462a37503078ff99e806d1f515516a482cff7ae0dea011b4aa5c80ffd707"} Nov 01 16:30:58.826052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.826029 4573 scope.go:122] "RemoveContainer" containerID="f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0" Nov 01 16:30:58.826341 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:58.826321 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0\": container with ID starting with f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0 not found: ID does not exist" containerID="f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0" Nov 01 16:30:58.826421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.826353 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0"} err="failed to get container status \"f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0\": rpc error: code = NotFound desc = could not find container \"f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0\": container with ID starting with f14a06c31237b4daf0c3aac87388a15a53867f22d9121262494c9e767830b9a0 not found: ID does not exist" Nov 01 16:30:58.826421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.826377 4573 scope.go:122] "RemoveContainer" containerID="f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc" Nov 01 16:30:58.837269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.837251 4573 scope.go:122] "RemoveContainer" containerID="f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc" Nov 01 16:30:58.837537 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:30:58.837514 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc\": container with ID starting with f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc not found: ID does not exist" containerID="f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc" Nov 01 16:30:58.837623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.837545 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc"} err="failed to get container status \"f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc\": rpc error: code = NotFound desc = could not find container \"f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc\": container with ID starting with f0668236f81eb834fc8d1d56bf46a53aa4f33b8e8818094e3e78a477e9998dbc not found: ID does not exist" Nov 01 16:30:58.840382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.840359 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g"] Nov 01 16:30:58.849483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.849463 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/simulator-w0-4a083d-kserve-686b95b847-zcq4g"] Nov 01 16:30:58.859678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.859647 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87"] Nov 01 16:30:58.863273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:58.863250 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-4a083d/premium-w0-4a083d-kserve-67695754f8-2mw87"] Nov 01 16:30:59.047111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:59.047023 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:30:59.938164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:30:59.938127 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:00.458925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:00.458892 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" path="/var/lib/kubelet/pods/2a7f0f44-adab-46e3-9232-a1a4f3a30e6e/volumes" Nov 01 16:31:00.459492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:00.459470 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" path="/var/lib/kubelet/pods/7062fcad-aa5b-4c09-b743-fbaa104863c6/volumes" Nov 01 16:31:00.655182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:00.655145 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:00.741123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:00.741030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:00.881774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:00.881283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:01.550909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.550870 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:01.895621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.895592 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:01.941329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.941296 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5"] Nov 01 16:31:01.942073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.942053 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" Nov 01 16:31:01.942073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.942074 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" Nov 01 16:31:01.942255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.942139 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" Nov 01 16:31:01.942255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.942147 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" Nov 01 16:31:01.942363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.942319 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a7f0f44-adab-46e3-9232-a1a4f3a30e6e" containerName="main" Nov 01 16:31:01.942363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.942339 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7062fcad-aa5b-4c09-b743-fbaa104863c6" containerName="main" Nov 01 16:31:01.947789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.947743 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:01.953221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.953197 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5"] Nov 01 16:31:01.954995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.954975 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-aa6303\"" Nov 01 16:31:01.955609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.955593 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-aa6303-dockercfg-82mtb\"" Nov 01 16:31:01.962285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.962262 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz"] Nov 01 16:31:01.966692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.966667 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:01.991407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.991374 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz"] Nov 01 16:31:01.999046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:01.999012 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:02.116772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.107919 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzjld\" (UniqueName: \"kubernetes.io/projected/abd82a21-2e85-4da0-92a1-ce251b547f49-kube-api-access-hzjld\") pod \"payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5\" (UID: \"abd82a21-2e85-4da0-92a1-ce251b547f49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:02.116772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.108048 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd82a21-2e85-4da0-92a1-ce251b547f49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5\" (UID: \"abd82a21-2e85-4da0-92a1-ce251b547f49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:02.116772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.108097 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgwtx\" (UniqueName: \"kubernetes.io/projected/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-kube-api-access-vgwtx\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz\" (UID: \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:02.116772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.108134 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz\" (UID: \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:02.211825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.210017 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz\" (UID: \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:02.211825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.210277 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzjld\" (UniqueName: \"kubernetes.io/projected/abd82a21-2e85-4da0-92a1-ce251b547f49-kube-api-access-hzjld\") pod \"payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5\" (UID: \"abd82a21-2e85-4da0-92a1-ce251b547f49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:02.211825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.210409 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd82a21-2e85-4da0-92a1-ce251b547f49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5\" (UID: \"abd82a21-2e85-4da0-92a1-ce251b547f49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:02.211825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.210459 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgwtx\" (UniqueName: \"kubernetes.io/projected/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-kube-api-access-vgwtx\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz\" (UID: \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:02.212184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.211968 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz\" (UID: \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:02.216487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.216274 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd82a21-2e85-4da0-92a1-ce251b547f49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5\" (UID: \"abd82a21-2e85-4da0-92a1-ce251b547f49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:02.238821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.238728 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzjld\" (UniqueName: \"kubernetes.io/projected/abd82a21-2e85-4da0-92a1-ce251b547f49-kube-api-access-hzjld\") pod \"payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5\" (UID: \"abd82a21-2e85-4da0-92a1-ce251b547f49\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:02.241460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.241427 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgwtx\" (UniqueName: \"kubernetes.io/projected/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-kube-api-access-vgwtx\") pod \"payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz\" (UID: \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:02.266990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.266964 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:02.293361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.293329 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:02.461997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.461831 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:02.581133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.581108 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5"] Nov 01 16:31:02.581562 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:31:02.581242 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd82a21_2e85_4da0_92a1_ce251b547f49.slice/crio-d5d61a91f33f693563b48ba8690bed5af252e7feef69b621cc8a35afa97fe8bd WatchSource:0}: Error finding container d5d61a91f33f693563b48ba8690bed5af252e7feef69b621cc8a35afa97fe8bd: Status 404 returned error can't find the container with id d5d61a91f33f693563b48ba8690bed5af252e7feef69b621cc8a35afa97fe8bd Nov 01 16:31:02.603810 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:31:02.603781 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c74843_b36d_4fc1_951b_1c4cedd9bfbd.slice/crio-11455b35dd356778ef9eb6c2d3ac25b7e3eb0ddaf1a6f519c70e41a339452c4e WatchSource:0}: Error finding container 11455b35dd356778ef9eb6c2d3ac25b7e3eb0ddaf1a6f519c70e41a339452c4e: Status 404 returned error can't find the container with id 11455b35dd356778ef9eb6c2d3ac25b7e3eb0ddaf1a6f519c70e41a339452c4e Nov 01 16:31:02.604315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.604210 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz"] Nov 01 16:31:02.899360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.899323 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" event={"ID":"98c74843-b36d-4fc1-951b-1c4cedd9bfbd","Type":"ContainerStarted","Data":"acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81"} Nov 01 16:31:02.899544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.899382 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" event={"ID":"98c74843-b36d-4fc1-951b-1c4cedd9bfbd","Type":"ContainerStarted","Data":"11455b35dd356778ef9eb6c2d3ac25b7e3eb0ddaf1a6f519c70e41a339452c4e"} Nov 01 16:31:02.899544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.899414 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:02.901249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.901216 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" event={"ID":"abd82a21-2e85-4da0-92a1-ce251b547f49","Type":"ContainerStarted","Data":"dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74"} Nov 01 16:31:02.901373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.901255 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" event={"ID":"abd82a21-2e85-4da0-92a1-ce251b547f49","Type":"ContainerStarted","Data":"d5d61a91f33f693563b48ba8690bed5af252e7feef69b621cc8a35afa97fe8bd"} Nov 01 16:31:02.901373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.901281 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:02.917730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.917681 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" podStartSLOduration=1.917669134 podStartE2EDuration="1.917669134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:31:02.913900502 +0000 UTC m=+2600.617066451" watchObservedRunningTime="2026-11-01 16:31:02.917669134 +0000 UTC m=+2600.620835070" Nov 01 16:31:02.934868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:02.934826 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" podStartSLOduration=1.9348137269999999 podStartE2EDuration="1.934813727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:31:02.934051442 +0000 UTC m=+2600.637217381" watchObservedRunningTime="2026-11-01 16:31:02.934813727 +0000 UTC m=+2600.637979662" Nov 01 16:31:03.277720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:03.277626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:03.960319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:03.960288 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:04.501086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:04.501054 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:31:04.695713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:04.693092 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:05.914777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:05.908800 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:06.576815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:06.576784 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:07.266529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:07.266490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:08.083331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:08.083300 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:08.782835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:08.782797 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:08.818128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:08.818104 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:31:09.659619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:09.659583 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:10.314302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:10.314275 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:11.024623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:11.024588 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:11.754354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:11.754320 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:11.988774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:11.988690 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:12.249431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.249398 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:12.296687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.296653 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8"] Nov 01 16:31:12.302306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.302279 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-4a083d-phlv8"] Nov 01 16:31:12.353350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.353321 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc"] Nov 01 16:31:12.357858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.357839 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.362197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.362171 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:31:12.362630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.362610 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:31:12.365026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.365005 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc"] Nov 01 16:31:12.384925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.384686 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:12.473398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.468862 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308c24e1-72df-4537-8453-24031a35c955" path="/var/lib/kubelet/pods/308c24e1-72df-4537-8453-24031a35c955/volumes" Nov 01 16:31:12.495826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.495767 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7hp\" (UniqueName: \"kubernetes.io/projected/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-kube-api-access-gb7hp\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.496007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.495852 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.496007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.495992 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.597438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.597345 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.597606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.597529 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7hp\" (UniqueName: \"kubernetes.io/projected/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-kube-api-access-gb7hp\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.597606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.597571 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.598213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.598189 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.599854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.599828 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.604987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.604957 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7hp\" (UniqueName: \"kubernetes.io/projected/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-kube-api-access-gb7hp\") pod \"maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.675496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.675473 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:12.780110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.780079 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:12.961038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:12.961012 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc"] Nov 01 16:31:12.961479 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:31:12.961461 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a5f15e1_e002_447e_a9b6_63dcc5e90bb8.slice/crio-03f31c2fecef7c62d66c05600b4c64bbea92a389ee9373a59d05d8809633a00f WatchSource:0}: Error finding container 03f31c2fecef7c62d66c05600b4c64bbea92a389ee9373a59d05d8809633a00f: Status 404 returned error can't find the container with id 03f31c2fecef7c62d66c05600b4c64bbea92a389ee9373a59d05d8809633a00f Nov 01 16:31:13.034824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:13.034778 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" event={"ID":"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8","Type":"ContainerStarted","Data":"03f31c2fecef7c62d66c05600b4c64bbea92a389ee9373a59d05d8809633a00f"} Nov 01 16:31:13.322858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:13.322826 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:13.911541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:13.911508 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:13.911967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:13.911575 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:14.045156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.045122 4573 generic.go:356] "Generic (PLEG): container finished" podID="8a5f15e1-e002-447e-a9b6-63dcc5e90bb8" containerID="233c3d5571dd7d52a155fc2341818557f0f3ba35ee71a5263333e5425c985944" exitCode=0 Nov 01 16:31:14.045325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.045168 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" event={"ID":"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8","Type":"ContainerDied","Data":"233c3d5571dd7d52a155fc2341818557f0f3ba35ee71a5263333e5425c985944"} Nov 01 16:31:14.215250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.215169 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:14.807690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.807659 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6"] Nov 01 16:31:14.812420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.812399 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:14.827954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.827155 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6"] Nov 01 16:31:14.838094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.838062 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-aa6303-serving-cert\"" Nov 01 16:31:14.845462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.845355 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:14.937777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.934968 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmsmd\" (UniqueName: \"kubernetes.io/projected/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-kube-api-access-pmsmd\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:14.937777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.935178 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:14.937777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.935286 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:14.937777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.935367 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:14.968410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:14.966294 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:15.036461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.036428 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.036672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.036514 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.036672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.036584 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmsmd\" (UniqueName: \"kubernetes.io/projected/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-kube-api-access-pmsmd\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.036830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.036770 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.037256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.037232 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.039033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.039004 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.039230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.039139 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-metrics-certs\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.044305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.044286 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmsmd\" (UniqueName: \"kubernetes.io/projected/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-kube-api-access-pmsmd\") pod \"maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.045863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.045835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:15.154524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.154488 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:15.245662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.243128 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:15.340730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.340702 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-kube-api-access-gb7hp\" (UniqueName: \"kubernetes.io/projected/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-kube-api-access-gb7hp\") pod \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " Nov 01 16:31:15.340932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.340799 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-openshift-ca\") pod \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " Nov 01 16:31:15.340932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.340831 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-xks-ca\") pod \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\" (UID: \"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8\") " Nov 01 16:31:15.341212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.341187 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-openshift-ca" pod "8a5f15e1-e002-447e-a9b6-63dcc5e90bb8" (UID: "8a5f15e1-e002-447e-a9b6-63dcc5e90bb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:15.341458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.341441 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:15.342969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.342946 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-xks-ca" pod "8a5f15e1-e002-447e-a9b6-63dcc5e90bb8" (UID: "8a5f15e1-e002-447e-a9b6-63dcc5e90bb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:15.343027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.342983 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-kube-api-access-gb7hp" pod "8a5f15e1-e002-447e-a9b6-63dcc5e90bb8" (UID: "8a5f15e1-e002-447e-a9b6-63dcc5e90bb8"). InnerVolumeSpecName "kube-api-access-gb7hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:15.446863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.446770 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:15.446863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.446813 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7hp\" (UniqueName: \"kubernetes.io/projected/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8-kube-api-access-gb7hp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:15.474048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.474023 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6"] Nov 01 16:31:15.474270 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:31:15.474246 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc61b142_ba1c_4fa4_b93f_95b880e2ae9b.slice/crio-3358e6ebbd99c748c3b266009c13880cac67009ebe66ed13905b3e2f1add0e28 WatchSource:0}: Error finding container 3358e6ebbd99c748c3b266009c13880cac67009ebe66ed13905b3e2f1add0e28: Status 404 returned error can't find the container with id 3358e6ebbd99c748c3b266009c13880cac67009ebe66ed13905b3e2f1add0e28 Nov 01 16:31:15.475727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.475709 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:31:15.713641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:15.713560 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:16.066945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:16.066906 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" event={"ID":"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b","Type":"ContainerStarted","Data":"fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67"} Nov 01 16:31:16.067800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:16.067015 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" event={"ID":"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b","Type":"ContainerStarted","Data":"3358e6ebbd99c748c3b266009c13880cac67009ebe66ed13905b3e2f1add0e28"} Nov 01 16:31:16.067800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:16.067262 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:16.069060 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:16.069041 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" Nov 01 16:31:16.069206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:16.069096 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc" event={"ID":"8a5f15e1-e002-447e-a9b6-63dcc5e90bb8","Type":"ContainerDied","Data":"03f31c2fecef7c62d66c05600b4c64bbea92a389ee9373a59d05d8809633a00f"} Nov 01 16:31:16.069206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:16.069144 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03f31c2fecef7c62d66c05600b4c64bbea92a389ee9373a59d05d8809633a00f" Nov 01 16:31:16.086530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:16.086485 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" podStartSLOduration=1.579222359 podStartE2EDuration="2.08647229s" totalImagesPullingTime="507.249931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:31:16.084089508 +0000 UTC m=+2613.787255450" watchObservedRunningTime="2026-11-01 16:31:16.08647229 +0000 UTC m=+2613.789638226" Nov 01 16:31:16.577041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:16.576998 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:17.290258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:17.290228 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:17.390695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:17.390658 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:18.084575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:18.084543 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:18.761410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:18.761373 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:18.878781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:18.878423 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:19.566721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:19.566689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:19.666672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:19.666637 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:19.782778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:19.782710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:19.987498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:19.987458 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b"] Nov 01 16:31:19.987732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:19.987701 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" containerID="cri-o://458e5d5b0e16192d433791b995be3e67b0a670f03dc195104d512a87c7e8e75a" gracePeriod=30 Nov 01 16:31:20.018705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:20.018674 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:20.394532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:20.394500 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:20.498373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:20.498341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:20.622372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:20.622316 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:20.872505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:20.872465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:21.366400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:21.366372 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:21.674810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:21.674691 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:22.082266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:22.082186 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:31:22.461935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:22.461899 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:23.151048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:23.151015 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:23.167044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:23.167012 4573 patch_prober.go:29] interesting pod/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:31:23.167199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:23.167063 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:31:24.056669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:24.056637 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:24.366413 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:31:24.366379 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:31:24.665177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:24.665097 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf"] Nov 01 16:31:24.665779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:24.665735 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" podUID="0118c8c5-5883-48ab-9457-ee52ec6b8f76" containerName="maas-api" containerID="cri-o://7a21ce54b2f2ad119971f005f4dabdb3e9313d293d6f0e6bd863ea089e0b1422" gracePeriod=30 Nov 01 16:31:24.692212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:24.692181 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:24.758811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:24.758779 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw"] Nov 01 16:31:24.759324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:24.759289 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" podUID="2753ad89-a39d-428b-ab90-c2923fd61ad9" containerName="payload-pre-processing" containerID="cri-o://8e43dc69345da5d0f6ac68d7e0cad274f3160432f3dd12f73e6c2aeedc48b651" gracePeriod=30 Nov 01 16:31:24.788482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:24.788452 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:24.862617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:24.862572 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:25.048859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.035661 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:25.094289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.080822 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2"] Nov 01 16:31:25.094289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.087368 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e96ce66-2gxk2"] Nov 01 16:31:25.191775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.180185 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:25.268776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.239615 4573 generic.go:356] "Generic (PLEG): container finished" podID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerID="458e5d5b0e16192d433791b995be3e67b0a670f03dc195104d512a87c7e8e75a" exitCode=0 Nov 01 16:31:25.268776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.239873 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" event={"ID":"c2b0a1ec-5862-4872-87cc-c6d5b040053c","Type":"ContainerDied","Data":"458e5d5b0e16192d433791b995be3e67b0a670f03dc195104d512a87c7e8e75a"} Nov 01 16:31:25.268776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.243212 4573 generic.go:356] "Generic (PLEG): container finished" podID="0118c8c5-5883-48ab-9457-ee52ec6b8f76" containerID="7a21ce54b2f2ad119971f005f4dabdb3e9313d293d6f0e6bd863ea089e0b1422" exitCode=0 Nov 01 16:31:25.268776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.243408 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" event={"ID":"0118c8c5-5883-48ab-9457-ee52ec6b8f76","Type":"ContainerDied","Data":"7a21ce54b2f2ad119971f005f4dabdb3e9313d293d6f0e6bd863ea089e0b1422"} Nov 01 16:31:25.268776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.245974 4573 generic.go:356] "Generic (PLEG): container finished" podID="2753ad89-a39d-428b-ab90-c2923fd61ad9" containerID="8e43dc69345da5d0f6ac68d7e0cad274f3160432f3dd12f73e6c2aeedc48b651" exitCode=0 Nov 01 16:31:25.268776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.246130 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" event={"ID":"2753ad89-a39d-428b-ab90-c2923fd61ad9","Type":"ContainerDied","Data":"8e43dc69345da5d0f6ac68d7e0cad274f3160432f3dd12f73e6c2aeedc48b651"} Nov 01 16:31:25.268776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.246937 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:31:25.414871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.414841 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-metrics-certs\") pod \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " Nov 01 16:31:25.415322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.414968 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-maas-api-tls\") pod \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " Nov 01 16:31:25.415322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.415075 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0118c8c5-5883-48ab-9457-ee52ec6b8f76-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0118c8c5-5883-48ab-9457-ee52ec6b8f76-openshift-service-ca\") pod \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " Nov 01 16:31:25.415322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.415131 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0118c8c5-5883-48ab-9457-ee52ec6b8f76-kube-api-access-tlbqz\" (UniqueName: \"kubernetes.io/projected/0118c8c5-5883-48ab-9457-ee52ec6b8f76-kube-api-access-tlbqz\") pod \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\" (UID: \"0118c8c5-5883-48ab-9457-ee52ec6b8f76\") " Nov 01 16:31:25.416334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.416309 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0118c8c5-5883-48ab-9457-ee52ec6b8f76-openshift-service-ca" pod "0118c8c5-5883-48ab-9457-ee52ec6b8f76" (UID: "0118c8c5-5883-48ab-9457-ee52ec6b8f76"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:25.417259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.417234 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:31:25.417368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.417283 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-metrics-certs" pod "0118c8c5-5883-48ab-9457-ee52ec6b8f76" (UID: "0118c8c5-5883-48ab-9457-ee52ec6b8f76"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:25.417545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.417523 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-maas-api-tls" pod "0118c8c5-5883-48ab-9457-ee52ec6b8f76" (UID: "0118c8c5-5883-48ab-9457-ee52ec6b8f76"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:25.417855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.417833 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0118c8c5-5883-48ab-9457-ee52ec6b8f76-kube-api-access-tlbqz" pod "0118c8c5-5883-48ab-9457-ee52ec6b8f76" (UID: "0118c8c5-5883-48ab-9457-ee52ec6b8f76"). InnerVolumeSpecName "kube-api-access-tlbqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:25.502145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.502096 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc"] Nov 01 16:31:25.502939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.502905 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:25.508207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.508182 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-3fd73f-tntbc"] Nov 01 16:31:25.517148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.517120 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2753ad89-a39d-428b-ab90-c2923fd61ad9-kube-api-access-fplrj\" (UniqueName: \"kubernetes.io/projected/2753ad89-a39d-428b-ab90-c2923fd61ad9-kube-api-access-fplrj\") pod \"2753ad89-a39d-428b-ab90-c2923fd61ad9\" (UID: \"2753ad89-a39d-428b-ab90-c2923fd61ad9\") " Nov 01 16:31:25.517291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.517276 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2753ad89-a39d-428b-ab90-c2923fd61ad9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753ad89-a39d-428b-ab90-c2923fd61ad9-plugins-config-volume\") pod \"2753ad89-a39d-428b-ab90-c2923fd61ad9\" (UID: \"2753ad89-a39d-428b-ab90-c2923fd61ad9\") " Nov 01 16:31:25.517698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.517651 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2753ad89-a39d-428b-ab90-c2923fd61ad9-plugins-config-volume" pod "2753ad89-a39d-428b-ab90-c2923fd61ad9" (UID: "2753ad89-a39d-428b-ab90-c2923fd61ad9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:25.518042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.518020 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.518165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.518046 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753ad89-a39d-428b-ab90-c2923fd61ad9-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.518165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.518060 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0118c8c5-5883-48ab-9457-ee52ec6b8f76-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.518165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.518072 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0118c8c5-5883-48ab-9457-ee52ec6b8f76-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.518165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.518086 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbqz\" (UniqueName: \"kubernetes.io/projected/0118c8c5-5883-48ab-9457-ee52ec6b8f76-kube-api-access-tlbqz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.520112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.520086 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2753ad89-a39d-428b-ab90-c2923fd61ad9-kube-api-access-fplrj" pod "2753ad89-a39d-428b-ab90-c2923fd61ad9" (UID: "2753ad89-a39d-428b-ab90-c2923fd61ad9"). InnerVolumeSpecName "kube-api-access-fplrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:25.576865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.576831 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:25.622474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.621844 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fplrj\" (UniqueName: \"kubernetes.io/projected/2753ad89-a39d-428b-ab90-c2923fd61ad9-kube-api-access-fplrj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.837294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.837228 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:31:25.925780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.925731 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-token\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-token\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.925954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.925811 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-podinfo\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.925954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.925854 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istiod-ca-cert\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.926056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926010 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-envoy\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.926231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926207 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-certs\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.926231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926238 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-kube-api-access-q7lc6\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-kube-api-access-q7lc6\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.926442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926259 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-credential-socket\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.926442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926255 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istiod-ca-cert" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:25.926442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926295 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-socket\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.926442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926396 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-data\") pod \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\" (UID: \"c2b0a1ec-5862-4872-87cc-c6d5b040053c\") " Nov 01 16:31:25.926669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926610 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-socket" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.926669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926627 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-credential-socket" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.926799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926688 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-certs" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.926954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.926933 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-data" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.927181 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.927165 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.927254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.927184 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.927254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.927193 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.927254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.927201 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.927254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.927210 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:25.928394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.928372 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-podinfo" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:31:25.928497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.928375 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-kube-api-access-q7lc6" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "kube-api-access-q7lc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:25.928595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.928575 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-token" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:25.928640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.928605 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-envoy" pod "c2b0a1ec-5862-4872-87cc-c6d5b040053c" (UID: "c2b0a1ec-5862-4872-87cc-c6d5b040053c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:31:25.967599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:25.967561 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:26.028699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.028666 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7lc6\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-kube-api-access-q7lc6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:26.028699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.028695 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:26.028699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.028705 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:26.028976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.028713 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2b0a1ec-5862-4872-87cc-c6d5b040053c-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:26.256320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.256288 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" Nov 01 16:31:26.256320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.256296 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b" event={"ID":"c2b0a1ec-5862-4872-87cc-c6d5b040053c","Type":"ContainerDied","Data":"b0c7562632797332df8b066257851784ffdd6a12461aa1f8b10bbfda25e73a86"} Nov 01 16:31:26.256608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.256362 4573 scope.go:122] "RemoveContainer" containerID="458e5d5b0e16192d433791b995be3e67b0a670f03dc195104d512a87c7e8e75a" Nov 01 16:31:26.258555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.258537 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" Nov 01 16:31:26.258658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.258587 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf" event={"ID":"0118c8c5-5883-48ab-9457-ee52ec6b8f76","Type":"ContainerDied","Data":"accef63858dc2742c0d91cb17f18b1e40c299c7aa551c3bc92e0f0cc29faaab2"} Nov 01 16:31:26.260516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.260487 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" event={"ID":"2753ad89-a39d-428b-ab90-c2923fd61ad9","Type":"ContainerDied","Data":"b96d19df608ad54429b74d0a726ec0a454f973bbd19135bebcd5a385a8cd5d42"} Nov 01 16:31:26.260704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.260678 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw" Nov 01 16:31:26.272863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.272720 4573 scope.go:122] "RemoveContainer" containerID="7a21ce54b2f2ad119971f005f4dabdb3e9313d293d6f0e6bd863ea089e0b1422" Nov 01 16:31:26.287519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.287490 4573 scope.go:122] "RemoveContainer" containerID="8e43dc69345da5d0f6ac68d7e0cad274f3160432f3dd12f73e6c2aeedc48b651" Nov 01 16:31:26.288382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.288357 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b"] Nov 01 16:31:26.297250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.296987 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-4a083d-openshift-default-888567bc7-bk68b"] Nov 01 16:31:26.302157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.302133 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf"] Nov 01 16:31:26.309575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.309558 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3fd73f-6c66487dfc-rxbrf"] Nov 01 16:31:26.318830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.318801 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw"] Nov 01 16:31:26.324043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.324019 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3fd73f-6f5d95576c-r9lrw"] Nov 01 16:31:26.395000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.394973 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:26.496691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.496660 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0118c8c5-5883-48ab-9457-ee52ec6b8f76" path="/var/lib/kubelet/pods/0118c8c5-5883-48ab-9457-ee52ec6b8f76/volumes" Nov 01 16:31:26.497170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.497155 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2753ad89-a39d-428b-ab90-c2923fd61ad9" path="/var/lib/kubelet/pods/2753ad89-a39d-428b-ab90-c2923fd61ad9/volumes" Nov 01 16:31:26.497579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.497566 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5f15e1-e002-447e-a9b6-63dcc5e90bb8" path="/var/lib/kubelet/pods/8a5f15e1-e002-447e-a9b6-63dcc5e90bb8/volumes" Nov 01 16:31:26.498391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.498373 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b986943c-9407-4ddd-b125-1f99a1ef2eb9" path="/var/lib/kubelet/pods/b986943c-9407-4ddd-b125-1f99a1ef2eb9/volumes" Nov 01 16:31:26.498956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:26.498937 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" path="/var/lib/kubelet/pods/c2b0a1ec-5862-4872-87cc-c6d5b040053c/volumes" Nov 01 16:31:27.219841 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:27.219807 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:28.053176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:28.053138 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:28.163189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:28.163154 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:28.856346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:28.856309 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:29.551917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.551886 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:29.721342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.721309 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl"] Nov 01 16:31:29.722016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722000 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2753ad89-a39d-428b-ab90-c2923fd61ad9" containerName="payload-pre-processing" Nov 01 16:31:29.722073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722017 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2753ad89-a39d-428b-ab90-c2923fd61ad9" containerName="payload-pre-processing" Nov 01 16:31:29.722073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722042 4573 cpu_manager.go:414] "containerMap: removing container" podUID="0118c8c5-5883-48ab-9457-ee52ec6b8f76" containerName="maas-api" Nov 01 16:31:29.722073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722048 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0118c8c5-5883-48ab-9457-ee52ec6b8f76" containerName="maas-api" Nov 01 16:31:29.722073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722067 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" Nov 01 16:31:29.722073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722071 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" Nov 01 16:31:29.722236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722085 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8a5f15e1-e002-447e-a9b6-63dcc5e90bb8" containerName="revoke-keys" Nov 01 16:31:29.722236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722089 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5f15e1-e002-447e-a9b6-63dcc5e90bb8" containerName="revoke-keys" Nov 01 16:31:29.722236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722175 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5f15e1-e002-447e-a9b6-63dcc5e90bb8" containerName="revoke-keys" Nov 01 16:31:29.722236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722188 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2753ad89-a39d-428b-ab90-c2923fd61ad9" containerName="payload-pre-processing" Nov 01 16:31:29.722236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722198 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="0118c8c5-5883-48ab-9457-ee52ec6b8f76" containerName="maas-api" Nov 01 16:31:29.722236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.722207 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b0a1ec-5862-4872-87cc-c6d5b040053c" containerName="istio-proxy" Nov 01 16:31:29.727916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.727888 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.732715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.732686 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:31:29.732916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.732717 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:31:29.756211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.756181 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl"] Nov 01 16:31:29.780631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.780600 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.780780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.780638 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.780850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.780803 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cfs\" (UniqueName: \"kubernetes.io/projected/f99db383-b057-446e-a54c-c5ed9a97aecd-kube-api-access-76cfs\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.881632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.881592 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.881850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.881639 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.881850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.881741 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cfs\" (UniqueName: \"kubernetes.io/projected/f99db383-b057-446e-a54c-c5ed9a97aecd-kube-api-access-76cfs\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.882475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.882451 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.884057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.884033 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:29.890978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:29.890958 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cfs\" (UniqueName: \"kubernetes.io/projected/f99db383-b057-446e-a54c-c5ed9a97aecd-kube-api-access-76cfs\") pod \"maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:30.057277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:30.057225 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:30.340055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:30.340030 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl"] Nov 01 16:31:30.340240 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:31:30.340218 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf99db383_b057_446e_a54c_c5ed9a97aecd.slice/crio-d09d86b2baec33ee43c4290566646376b63d6acdb2ad064f6f46c8f454e8b538 WatchSource:0}: Error finding container d09d86b2baec33ee43c4290566646376b63d6acdb2ad064f6f46c8f454e8b538: Status 404 returned error can't find the container with id d09d86b2baec33ee43c4290566646376b63d6acdb2ad064f6f46c8f454e8b538 Nov 01 16:31:30.631775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:30.630653 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:31.394320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:31.394271 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" event={"ID":"f99db383-b057-446e-a54c-c5ed9a97aecd","Type":"ContainerStarted","Data":"68a04ebbfb1497fec54c0c8b2492118df81cca56f00720bce64a4aee62a84d99"} Nov 01 16:31:31.394636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:31.394595 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" event={"ID":"f99db383-b057-446e-a54c-c5ed9a97aecd","Type":"ContainerStarted","Data":"d09d86b2baec33ee43c4290566646376b63d6acdb2ad064f6f46c8f454e8b538"} Nov 01 16:31:31.423708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:31.418710 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" podStartSLOduration=2.418695243 podStartE2EDuration="2.418695243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:31:31.416572462 +0000 UTC m=+2629.119738399" watchObservedRunningTime="2026-11-01 16:31:31.418695243 +0000 UTC m=+2629.121861177" Nov 01 16:31:31.473087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:31.473058 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:32.287099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:32.287066 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:32.388172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:32.388132 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:32.404047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:32.404015 4573 generic.go:356] "Generic (PLEG): container finished" podID="f99db383-b057-446e-a54c-c5ed9a97aecd" containerID="68a04ebbfb1497fec54c0c8b2492118df81cca56f00720bce64a4aee62a84d99" exitCode=0 Nov 01 16:31:32.404240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:32.404100 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" event={"ID":"f99db383-b057-446e-a54c-c5ed9a97aecd","Type":"ContainerDied","Data":"68a04ebbfb1497fec54c0c8b2492118df81cca56f00720bce64a4aee62a84d99"} Nov 01 16:31:32.564992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:32.564864 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:32.649019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:32.648984 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:33.458444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.458392 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:33.590932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.590906 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:33.648714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.648685 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f99db383-b057-446e-a54c-c5ed9a97aecd-kube-api-access-76cfs\" (UniqueName: \"kubernetes.io/projected/f99db383-b057-446e-a54c-c5ed9a97aecd-kube-api-access-76cfs\") pod \"f99db383-b057-446e-a54c-c5ed9a97aecd\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " Nov 01 16:31:33.648920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.648779 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-openshift-ca\") pod \"f99db383-b057-446e-a54c-c5ed9a97aecd\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " Nov 01 16:31:33.648920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.648867 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-xks-ca\") pod \"f99db383-b057-446e-a54c-c5ed9a97aecd\" (UID: \"f99db383-b057-446e-a54c-c5ed9a97aecd\") " Nov 01 16:31:33.649161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.649138 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-openshift-ca" pod "f99db383-b057-446e-a54c-c5ed9a97aecd" (UID: "f99db383-b057-446e-a54c-c5ed9a97aecd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:33.649547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.649529 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:33.650975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.650944 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f99db383-b057-446e-a54c-c5ed9a97aecd-kube-api-access-76cfs" pod "f99db383-b057-446e-a54c-c5ed9a97aecd" (UID: "f99db383-b057-446e-a54c-c5ed9a97aecd"). InnerVolumeSpecName "kube-api-access-76cfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:33.651081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.650947 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-xks-ca" pod "f99db383-b057-446e-a54c-c5ed9a97aecd" (UID: "f99db383-b057-446e-a54c-c5ed9a97aecd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:33.751099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.751027 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f99db383-b057-446e-a54c-c5ed9a97aecd-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:33.751099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:33.751062 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76cfs\" (UniqueName: \"kubernetes.io/projected/f99db383-b057-446e-a54c-c5ed9a97aecd-kube-api-access-76cfs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:34.345677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:34.345648 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:34.440919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:34.440877 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" event={"ID":"f99db383-b057-446e-a54c-c5ed9a97aecd","Type":"ContainerDied","Data":"d09d86b2baec33ee43c4290566646376b63d6acdb2ad064f6f46c8f454e8b538"} Nov 01 16:31:34.441131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:34.440943 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl" Nov 01 16:31:34.441602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:34.440941 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d09d86b2baec33ee43c4290566646376b63d6acdb2ad064f6f46c8f454e8b538" Nov 01 16:31:34.978825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:34.978795 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:35.577575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:35.577542 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:35.682989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:35.682952 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:36.025271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.025231 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2"] Nov 01 16:31:36.029100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.029070 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0635104d-4f9z2"] Nov 01 16:31:36.063336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.063307 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5"] Nov 01 16:31:36.077065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.077033 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz"] Nov 01 16:31:36.125438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.125402 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:36.227417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.227377 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:36.392787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.390259 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:36.468427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.468387 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce15495c-2781-485a-b362-7c71f5540a67" path="/var/lib/kubelet/pods/ce15495c-2781-485a-b362-7c71f5540a67/volumes" Nov 01 16:31:36.470698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:36.470673 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:37.462232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:37.462173 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:38.264382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:38.264353 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:38.889344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:38.889306 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:39.649712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:39.649676 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:40.288407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:40.288370 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:41.064165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.064112 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" podUID="abd82a21-2e85-4da0-92a1-ce251b547f49" containerName="payload-processing" containerID="cri-o://dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74" gracePeriod=25 Nov 01 16:31:41.072204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.072174 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:41.078231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.078201 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" podUID="98c74843-b36d-4fc1-951b-1c4cedd9bfbd" containerName="payload-pre-processing" containerID="cri-o://acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81" gracePeriod=25 Nov 01 16:31:41.365910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.365881 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:41.395710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.395681 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd82a21-2e85-4da0-92a1-ce251b547f49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd82a21-2e85-4da0-92a1-ce251b547f49-plugins-config-volume\") pod \"abd82a21-2e85-4da0-92a1-ce251b547f49\" (UID: \"abd82a21-2e85-4da0-92a1-ce251b547f49\") " Nov 01 16:31:41.395901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.395883 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd82a21-2e85-4da0-92a1-ce251b547f49-kube-api-access-hzjld\" (UniqueName: \"kubernetes.io/projected/abd82a21-2e85-4da0-92a1-ce251b547f49-kube-api-access-hzjld\") pod \"abd82a21-2e85-4da0-92a1-ce251b547f49\" (UID: \"abd82a21-2e85-4da0-92a1-ce251b547f49\") " Nov 01 16:31:41.396077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.396054 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd82a21-2e85-4da0-92a1-ce251b547f49-plugins-config-volume" pod "abd82a21-2e85-4da0-92a1-ce251b547f49" (UID: "abd82a21-2e85-4da0-92a1-ce251b547f49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:41.396486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.396469 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd82a21-2e85-4da0-92a1-ce251b547f49-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:41.398269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.398242 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd82a21-2e85-4da0-92a1-ce251b547f49-kube-api-access-hzjld" pod "abd82a21-2e85-4da0-92a1-ce251b547f49" (UID: "abd82a21-2e85-4da0-92a1-ce251b547f49"). InnerVolumeSpecName "kube-api-access-hzjld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:41.399265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.399243 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:41.497074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.497041 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-kube-api-access-vgwtx\" (UniqueName: \"kubernetes.io/projected/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-kube-api-access-vgwtx\") pod \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\" (UID: \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\") " Nov 01 16:31:41.497276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.497145 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-plugins-config-volume\") pod \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\" (UID: \"98c74843-b36d-4fc1-951b-1c4cedd9bfbd\") " Nov 01 16:31:41.497582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.497551 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-plugins-config-volume" pod "98c74843-b36d-4fc1-951b-1c4cedd9bfbd" (UID: "98c74843-b36d-4fc1-951b-1c4cedd9bfbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:41.497713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.497580 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzjld\" (UniqueName: \"kubernetes.io/projected/abd82a21-2e85-4da0-92a1-ce251b547f49-kube-api-access-hzjld\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:41.499449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.499421 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-kube-api-access-vgwtx" pod "98c74843-b36d-4fc1-951b-1c4cedd9bfbd" (UID: "98c74843-b36d-4fc1-951b-1c4cedd9bfbd"). InnerVolumeSpecName "kube-api-access-vgwtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:41.500405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.500379 4573 generic.go:356] "Generic (PLEG): container finished" podID="abd82a21-2e85-4da0-92a1-ce251b547f49" containerID="dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74" exitCode=0 Nov 01 16:31:41.500503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.500429 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" Nov 01 16:31:41.500503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.500460 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" event={"ID":"abd82a21-2e85-4da0-92a1-ce251b547f49","Type":"ContainerDied","Data":"dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74"} Nov 01 16:31:41.500596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.500517 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5" event={"ID":"abd82a21-2e85-4da0-92a1-ce251b547f49","Type":"ContainerDied","Data":"d5d61a91f33f693563b48ba8690bed5af252e7feef69b621cc8a35afa97fe8bd"} Nov 01 16:31:41.500596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.500543 4573 scope.go:122] "RemoveContainer" containerID="dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74" Nov 01 16:31:41.502176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.502156 4573 generic.go:356] "Generic (PLEG): container finished" podID="98c74843-b36d-4fc1-951b-1c4cedd9bfbd" containerID="acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81" exitCode=0 Nov 01 16:31:41.502286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.502207 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" Nov 01 16:31:41.502286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.502213 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" event={"ID":"98c74843-b36d-4fc1-951b-1c4cedd9bfbd","Type":"ContainerDied","Data":"acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81"} Nov 01 16:31:41.502286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.502257 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz" event={"ID":"98c74843-b36d-4fc1-951b-1c4cedd9bfbd","Type":"ContainerDied","Data":"11455b35dd356778ef9eb6c2d3ac25b7e3eb0ddaf1a6f519c70e41a339452c4e"} Nov 01 16:31:41.518403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.518381 4573 scope.go:122] "RemoveContainer" containerID="dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74" Nov 01 16:31:41.518738 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:31:41.518716 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74\": container with ID starting with dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74 not found: ID does not exist" containerID="dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74" Nov 01 16:31:41.518880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.518760 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74"} err="failed to get container status \"dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74\": rpc error: code = NotFound desc = could not find container \"dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74\": container with ID starting with dd44523b7bad1aa6de9d1724a7590096d7ccf785d676dd4964d7a2e692183d74 not found: ID does not exist" Nov 01 16:31:41.518880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.518787 4573 scope.go:122] "RemoveContainer" containerID="acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81" Nov 01 16:31:41.529968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.529937 4573 scope.go:122] "RemoveContainer" containerID="acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81" Nov 01 16:31:41.530250 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:31:41.530231 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81\": container with ID starting with acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81 not found: ID does not exist" containerID="acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81" Nov 01 16:31:41.530312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.530259 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81"} err="failed to get container status \"acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81\": rpc error: code = NotFound desc = could not find container \"acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81\": container with ID starting with acd355904cad4d5977e411b322e33d7bdcfa6aec44eca35ed0d8905845607a81 not found: ID does not exist" Nov 01 16:31:41.532057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.532038 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5"] Nov 01 16:31:41.537761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.537728 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-aa6303-59d6d99bc5-lqbj5"] Nov 01 16:31:41.547335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.547312 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz"] Nov 01 16:31:41.551862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.551838 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-aa6303-667996645f-xzklz"] Nov 01 16:31:41.599193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.599123 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:41.599193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:41.599150 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgwtx\" (UniqueName: \"kubernetes.io/projected/98c74843-b36d-4fc1-951b-1c4cedd9bfbd-kube-api-access-vgwtx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:42.159614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:42.159576 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:42.465290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:42.465255 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c74843-b36d-4fc1-951b-1c4cedd9bfbd" path="/var/lib/kubelet/pods/98c74843-b36d-4fc1-951b-1c4cedd9bfbd/volumes" Nov 01 16:31:42.465923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:42.465906 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd82a21-2e85-4da0-92a1-ce251b547f49" path="/var/lib/kubelet/pods/abd82a21-2e85-4da0-92a1-ce251b547f49/volumes" Nov 01 16:31:43.156437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:43.156405 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:43.865897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:43.865863 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:44.541865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:44.541834 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:45.239314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:45.239283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:45.721317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:45.721283 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:45.819666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:45.819628 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:46.370470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.370433 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:46.503701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.503672 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk"] Nov 01 16:31:46.504331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504316 4573 cpu_manager.go:414] "containerMap: removing container" podUID="98c74843-b36d-4fc1-951b-1c4cedd9bfbd" containerName="payload-pre-processing" Nov 01 16:31:46.504395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504333 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c74843-b36d-4fc1-951b-1c4cedd9bfbd" containerName="payload-pre-processing" Nov 01 16:31:46.504395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504350 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f99db383-b057-446e-a54c-c5ed9a97aecd" containerName="revoke-keys" Nov 01 16:31:46.504395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504356 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f99db383-b057-446e-a54c-c5ed9a97aecd" containerName="revoke-keys" Nov 01 16:31:46.504395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504378 4573 cpu_manager.go:414] "containerMap: removing container" podUID="abd82a21-2e85-4da0-92a1-ce251b547f49" containerName="payload-processing" Nov 01 16:31:46.504395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504383 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd82a21-2e85-4da0-92a1-ce251b547f49" containerName="payload-processing" Nov 01 16:31:46.504587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504479 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd82a21-2e85-4da0-92a1-ce251b547f49" containerName="payload-processing" Nov 01 16:31:46.504587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504492 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c74843-b36d-4fc1-951b-1c4cedd9bfbd" containerName="payload-pre-processing" Nov 01 16:31:46.504587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.504499 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f99db383-b057-446e-a54c-c5ed9a97aecd" containerName="revoke-keys" Nov 01 16:31:46.509702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.509681 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk"] Nov 01 16:31:46.509844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.509802 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.514113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.514090 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:31:46.514264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.514109 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:31:46.590967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.590935 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tpmg\" (UniqueName: \"kubernetes.io/projected/6d70c578-0744-48fb-9851-012201fbbc20-kube-api-access-7tpmg\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.591132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.591028 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.591132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.591093 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.692708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.692630 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tpmg\" (UniqueName: \"kubernetes.io/projected/6d70c578-0744-48fb-9851-012201fbbc20-kube-api-access-7tpmg\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.692708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.692685 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.692931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.692715 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.693447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.693427 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.695176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.695149 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.700100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.700077 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tpmg\" (UniqueName: \"kubernetes.io/projected/6d70c578-0744-48fb-9851-012201fbbc20-kube-api-access-7tpmg\") pod \"maas-api-revoke-sub-27fd13bd-8vblk\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:46.828616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:46.828586 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:47.120189 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:31:47.120159 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d70c578_0744_48fb_9851_012201fbbc20.slice/crio-8fe6f5e61eafec76da709017c7d96bd16edc5706845f98a11ab831f900739bf3 WatchSource:0}: Error finding container 8fe6f5e61eafec76da709017c7d96bd16edc5706845f98a11ab831f900739bf3: Status 404 returned error can't find the container with id 8fe6f5e61eafec76da709017c7d96bd16edc5706845f98a11ab831f900739bf3 Nov 01 16:31:47.120389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:47.120368 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk"] Nov 01 16:31:47.160461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:47.160438 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:47.555891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:47.555855 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" event={"ID":"6d70c578-0744-48fb-9851-012201fbbc20","Type":"ContainerStarted","Data":"29c0812067cba1cf270891c0dbb03d22da7f14d74a61c7e638d55e31b98a7000"} Nov 01 16:31:47.556313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:47.555949 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" event={"ID":"6d70c578-0744-48fb-9851-012201fbbc20","Type":"ContainerStarted","Data":"8fe6f5e61eafec76da709017c7d96bd16edc5706845f98a11ab831f900739bf3"} Nov 01 16:31:47.574792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:47.574728 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" podStartSLOduration=1.574714246 podStartE2EDuration="1.574714246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:31:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:31:47.570717044 +0000 UTC m=+2645.273882981" watchObservedRunningTime="2026-11-01 16:31:47.574714246 +0000 UTC m=+2645.277880181" Nov 01 16:31:47.881794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:47.881745 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:48.551574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:48.551537 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:48.565212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:48.565182 4573 generic.go:356] "Generic (PLEG): container finished" podID="6d70c578-0744-48fb-9851-012201fbbc20" containerID="29c0812067cba1cf270891c0dbb03d22da7f14d74a61c7e638d55e31b98a7000" exitCode=0 Nov 01 16:31:48.565633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:48.565268 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" event={"ID":"6d70c578-0744-48fb-9851-012201fbbc20","Type":"ContainerDied","Data":"29c0812067cba1cf270891c0dbb03d22da7f14d74a61c7e638d55e31b98a7000"} Nov 01 16:31:49.180986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.180947 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:49.219671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.219645 4573 scope.go:122] "RemoveContainer" containerID="5f88c347dc06fcb05bb509c6271219d9dd8e1f0a046ad11fd844b4865492b459" Nov 01 16:31:49.258976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.254866 4573 scope.go:122] "RemoveContainer" containerID="07473e4cdbff615c5a4e13021d574d379371cd1d7d1ac38ba33c4e49b2101cd2" Nov 01 16:31:49.748604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.748578 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:49.861022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.860992 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-xks-ca\") pod \"6d70c578-0744-48fb-9851-012201fbbc20\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " Nov 01 16:31:49.861270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.861254 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d70c578-0744-48fb-9851-012201fbbc20-kube-api-access-7tpmg\" (UniqueName: \"kubernetes.io/projected/6d70c578-0744-48fb-9851-012201fbbc20-kube-api-access-7tpmg\") pod \"6d70c578-0744-48fb-9851-012201fbbc20\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " Nov 01 16:31:49.861357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.861290 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-openshift-ca\") pod \"6d70c578-0744-48fb-9851-012201fbbc20\" (UID: \"6d70c578-0744-48fb-9851-012201fbbc20\") " Nov 01 16:31:49.862031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.862000 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-openshift-ca" pod "6d70c578-0744-48fb-9851-012201fbbc20" (UID: "6d70c578-0744-48fb-9851-012201fbbc20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:49.863158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.863131 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-xks-ca" pod "6d70c578-0744-48fb-9851-012201fbbc20" (UID: "6d70c578-0744-48fb-9851-012201fbbc20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:49.863335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.863319 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d70c578-0744-48fb-9851-012201fbbc20-kube-api-access-7tpmg" pod "6d70c578-0744-48fb-9851-012201fbbc20" (UID: "6d70c578-0744-48fb-9851-012201fbbc20"). InnerVolumeSpecName "kube-api-access-7tpmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:49.927672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.927620 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:49.963362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.963334 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tpmg\" (UniqueName: \"kubernetes.io/projected/6d70c578-0744-48fb-9851-012201fbbc20-kube-api-access-7tpmg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:49.963362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.963363 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:49.963545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:49.963375 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d70c578-0744-48fb-9851-012201fbbc20-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:50.438320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.438281 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt"] Nov 01 16:31:50.438573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.438522 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" podUID="470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" containerName="maas-api" containerID="cri-o://79f85191a88e05d9fe74c332333453e1415188a41ef14c559c3d46a632968c97" gracePeriod=30 Nov 01 16:31:50.478709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.478682 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:50.518038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.518002 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l"] Nov 01 16:31:50.518492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.518463 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" podUID="d866f151-f5b8-4f65-b376-24e6c0da561e" containerName="payload-processing" containerID="cri-o://63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea" gracePeriod=30 Nov 01 16:31:50.578221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.578186 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:50.591516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.591486 4573 generic.go:356] "Generic (PLEG): container finished" podID="470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" containerID="79f85191a88e05d9fe74c332333453e1415188a41ef14c559c3d46a632968c97" exitCode=0 Nov 01 16:31:50.591690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.591596 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" event={"ID":"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e","Type":"ContainerDied","Data":"79f85191a88e05d9fe74c332333453e1415188a41ef14c559c3d46a632968c97"} Nov 01 16:31:50.593397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.593374 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" event={"ID":"6d70c578-0744-48fb-9851-012201fbbc20","Type":"ContainerDied","Data":"8fe6f5e61eafec76da709017c7d96bd16edc5706845f98a11ab831f900739bf3"} Nov 01 16:31:50.593519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.593418 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fe6f5e61eafec76da709017c7d96bd16edc5706845f98a11ab831f900739bf3" Nov 01 16:31:50.593519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.593490 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk" Nov 01 16:31:50.677556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.677513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:50.856935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.855653 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:50.901269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.901231 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl"] Nov 01 16:31:50.905230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.905204 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2b94b5-hfnvl"] Nov 01 16:31:50.931510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.931468 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:50.971236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.971213 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:31:50.999206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.997912 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-kube-api-access-wc6g2\" (UniqueName: \"kubernetes.io/projected/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-kube-api-access-wc6g2\") pod \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " Nov 01 16:31:50.999206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.998073 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-openshift-service-ca\") pod \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " Nov 01 16:31:50.999206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.998179 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-metrics-certs\") pod \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " Nov 01 16:31:50.999206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.998226 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-maas-api-tls\") pod \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\" (UID: \"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e\") " Nov 01 16:31:50.999828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:50.999727 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-openshift-service-ca" pod "470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" (UID: "470a19cf-a76f-46dc-ae6b-9d9c6cd9163e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:51.007545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.007506 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-maas-api-tls" pod "470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" (UID: "470a19cf-a76f-46dc-ae6b-9d9c6cd9163e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:51.012377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.012351 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-metrics-certs" pod "470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" (UID: "470a19cf-a76f-46dc-ae6b-9d9c6cd9163e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:51.012592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.012405 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-kube-api-access-wc6g2" pod "470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" (UID: "470a19cf-a76f-46dc-ae6b-9d9c6cd9163e"). InnerVolumeSpecName "kube-api-access-wc6g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:51.013298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.013256 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:31:51.100334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.100295 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d866f151-f5b8-4f65-b376-24e6c0da561e-kube-api-access-tx65m\" (UniqueName: \"kubernetes.io/projected/d866f151-f5b8-4f65-b376-24e6c0da561e-kube-api-access-tx65m\") pod \"d866f151-f5b8-4f65-b376-24e6c0da561e\" (UID: \"d866f151-f5b8-4f65-b376-24e6c0da561e\") " Nov 01 16:31:51.100571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.100405 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d866f151-f5b8-4f65-b376-24e6c0da561e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d866f151-f5b8-4f65-b376-24e6c0da561e-plugins-config-volume\") pod \"d866f151-f5b8-4f65-b376-24e6c0da561e\" (UID: \"d866f151-f5b8-4f65-b376-24e6c0da561e\") " Nov 01 16:31:51.100909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.100872 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d866f151-f5b8-4f65-b376-24e6c0da561e-plugins-config-volume" pod "d866f151-f5b8-4f65-b376-24e6c0da561e" (UID: "d866f151-f5b8-4f65-b376-24e6c0da561e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:51.101147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.101122 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d866f151-f5b8-4f65-b376-24e6c0da561e-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:51.101243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.101154 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:51.101243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.101169 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:51.101243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.101184 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:51.101243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.101198 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc6g2\" (UniqueName: \"kubernetes.io/projected/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e-kube-api-access-wc6g2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:51.102564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.102539 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d866f151-f5b8-4f65-b376-24e6c0da561e-kube-api-access-tx65m" pod "d866f151-f5b8-4f65-b376-24e6c0da561e" (UID: "d866f151-f5b8-4f65-b376-24e6c0da561e"). InnerVolumeSpecName "kube-api-access-tx65m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:51.203077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.203039 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx65m\" (UniqueName: \"kubernetes.io/projected/d866f151-f5b8-4f65-b376-24e6c0da561e-kube-api-access-tx65m\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:51.367351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.367301 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:51.601344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.601320 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" Nov 01 16:31:51.601562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.601365 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt" event={"ID":"470a19cf-a76f-46dc-ae6b-9d9c6cd9163e","Type":"ContainerDied","Data":"8736c2f8943107bd3beb59008bb87e02555a52253ef3605b1936077e2572ed7e"} Nov 01 16:31:51.601562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.601433 4573 scope.go:122] "RemoveContainer" containerID="79f85191a88e05d9fe74c332333453e1415188a41ef14c559c3d46a632968c97" Nov 01 16:31:51.602834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.602742 4573 generic.go:356] "Generic (PLEG): container finished" podID="d866f151-f5b8-4f65-b376-24e6c0da561e" containerID="63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea" exitCode=0 Nov 01 16:31:51.602834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.602825 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" Nov 01 16:31:51.602944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.602858 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" event={"ID":"d866f151-f5b8-4f65-b376-24e6c0da561e","Type":"ContainerDied","Data":"63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea"} Nov 01 16:31:51.602944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.602907 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l" event={"ID":"d866f151-f5b8-4f65-b376-24e6c0da561e","Type":"ContainerDied","Data":"de94dd2196e0ed6fd390bbdb1ecd42556a130cab680435510358c5c44187f7c3"} Nov 01 16:31:51.614134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.614044 4573 scope.go:122] "RemoveContainer" containerID="63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea" Nov 01 16:31:51.628868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.628850 4573 scope.go:122] "RemoveContainer" containerID="63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea" Nov 01 16:31:51.629211 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:31:51.629186 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea\": container with ID starting with 63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea not found: ID does not exist" containerID="63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea" Nov 01 16:31:51.629315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.629217 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea"} err="failed to get container status \"63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea\": rpc error: code = NotFound desc = could not find container \"63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea\": container with ID starting with 63072e1bcbb684782c6021883f3e9c54e1bff5afbb2085521d8db510fc49e6ea not found: ID does not exist" Nov 01 16:31:51.635077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.635056 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt"] Nov 01 16:31:51.638677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.638656 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2b94b5-db5df755f-rnsvt"] Nov 01 16:31:51.647099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.647077 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l"] Nov 01 16:31:51.651120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.651095 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2b94b5-75d8b89bfb-7kk7l"] Nov 01 16:31:51.869588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:51.869549 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:52.458731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:52.458696 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" path="/var/lib/kubelet/pods/470a19cf-a76f-46dc-ae6b-9d9c6cd9163e/volumes" Nov 01 16:31:52.459162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:52.459146 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d866f151-f5b8-4f65-b376-24e6c0da561e" path="/var/lib/kubelet/pods/d866f151-f5b8-4f65-b376-24e6c0da561e/volumes" Nov 01 16:31:52.459546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:52.459533 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f99db383-b057-446e-a54c-c5ed9a97aecd" path="/var/lib/kubelet/pods/f99db383-b057-446e-a54c-c5ed9a97aecd/volumes" Nov 01 16:31:52.563924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:52.563890 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:53.496045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:53.496010 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:54.185160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:54.185111 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:54.366630 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:31:54.366595 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:31:54.847159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:54.847130 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:55.407449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.407419 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:55.440991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.440960 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv"] Nov 01 16:31:55.441207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.441176 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" podUID="803fc24d-d984-49ee-a476-c6dbb72aa92e" containerName="payload-processing" containerID="cri-o://c2cebf96215f5529808ad9abb7c27507f38a2c3d7df461ab8ccd8809c45aa494" gracePeriod=30 Nov 01 16:31:55.505259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.505222 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:55.607857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.607825 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:55.650286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.650257 4573 generic.go:356] "Generic (PLEG): container finished" podID="803fc24d-d984-49ee-a476-c6dbb72aa92e" containerID="c2cebf96215f5529808ad9abb7c27507f38a2c3d7df461ab8ccd8809c45aa494" exitCode=0 Nov 01 16:31:55.650444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.650297 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" event={"ID":"803fc24d-d984-49ee-a476-c6dbb72aa92e","Type":"ContainerDied","Data":"c2cebf96215f5529808ad9abb7c27507f38a2c3d7df461ab8ccd8809c45aa494"} Nov 01 16:31:55.692635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.692576 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:55.713920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.713896 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:31:55.766157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.766112 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:55.847788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.841278 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803fc24d-d984-49ee-a476-c6dbb72aa92e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803fc24d-d984-49ee-a476-c6dbb72aa92e-plugins-config-volume\") pod \"803fc24d-d984-49ee-a476-c6dbb72aa92e\" (UID: \"803fc24d-d984-49ee-a476-c6dbb72aa92e\") " Nov 01 16:31:55.847788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.841490 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803fc24d-d984-49ee-a476-c6dbb72aa92e-kube-api-access-tvmg2\" (UniqueName: \"kubernetes.io/projected/803fc24d-d984-49ee-a476-c6dbb72aa92e-kube-api-access-tvmg2\") pod \"803fc24d-d984-49ee-a476-c6dbb72aa92e\" (UID: \"803fc24d-d984-49ee-a476-c6dbb72aa92e\") " Nov 01 16:31:55.877619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.861863 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803fc24d-d984-49ee-a476-c6dbb72aa92e-plugins-config-volume" pod "803fc24d-d984-49ee-a476-c6dbb72aa92e" (UID: "803fc24d-d984-49ee-a476-c6dbb72aa92e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:55.877619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.864858 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803fc24d-d984-49ee-a476-c6dbb72aa92e-kube-api-access-tvmg2" pod "803fc24d-d984-49ee-a476-c6dbb72aa92e" (UID: "803fc24d-d984-49ee-a476-c6dbb72aa92e"). InnerVolumeSpecName "kube-api-access-tvmg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:31:55.943283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.943202 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmg2\" (UniqueName: \"kubernetes.io/projected/803fc24d-d984-49ee-a476-c6dbb72aa92e-kube-api-access-tvmg2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:55.943283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:55.943232 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803fc24d-d984-49ee-a476-c6dbb72aa92e-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:56.138505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.138466 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:56.671288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.671247 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" Nov 01 16:31:56.671594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.671249 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv" event={"ID":"803fc24d-d984-49ee-a476-c6dbb72aa92e","Type":"ContainerDied","Data":"2cac6a455c44ef1b1f2eaac97876d1d57518d089f61593149f16f67f8227fbf0"} Nov 01 16:31:56.671594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.671397 4573 scope.go:122] "RemoveContainer" containerID="c2cebf96215f5529808ad9abb7c27507f38a2c3d7df461ab8ccd8809c45aa494" Nov 01 16:31:56.683380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.683352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:56.692784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.692744 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv"] Nov 01 16:31:56.700263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.700241 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-6b6cfc-67879db544-brnzv"] Nov 01 16:31:56.853134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853057 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d"] Nov 01 16:31:56.853696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853682 4573 cpu_manager.go:414] "containerMap: removing container" podUID="803fc24d-d984-49ee-a476-c6dbb72aa92e" containerName="payload-processing" Nov 01 16:31:56.853746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853697 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803fc24d-d984-49ee-a476-c6dbb72aa92e" containerName="payload-processing" Nov 01 16:31:56.853746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853714 4573 cpu_manager.go:414] "containerMap: removing container" podUID="470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" containerName="maas-api" Nov 01 16:31:56.853746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853719 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" containerName="maas-api" Nov 01 16:31:56.853746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853743 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6d70c578-0744-48fb-9851-012201fbbc20" containerName="revoke-keys" Nov 01 16:31:56.853746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853763 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d70c578-0744-48fb-9851-012201fbbc20" containerName="revoke-keys" Nov 01 16:31:56.853746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853773 4573 cpu_manager.go:414] "containerMap: removing container" podUID="d866f151-f5b8-4f65-b376-24e6c0da561e" containerName="payload-processing" Nov 01 16:31:56.853746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853778 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d866f151-f5b8-4f65-b376-24e6c0da561e" containerName="payload-processing" Nov 01 16:31:56.854042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853875 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="803fc24d-d984-49ee-a476-c6dbb72aa92e" containerName="payload-processing" Nov 01 16:31:56.854042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853885 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="470a19cf-a76f-46dc-ae6b-9d9c6cd9163e" containerName="maas-api" Nov 01 16:31:56.854042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853896 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d70c578-0744-48fb-9851-012201fbbc20" containerName="revoke-keys" Nov 01 16:31:56.854042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.853902 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="d866f151-f5b8-4f65-b376-24e6c0da561e" containerName="payload-processing" Nov 01 16:31:56.859218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.859200 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:56.862850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.862804 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:31:56.862997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.862965 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:31:56.866195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.866169 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d"] Nov 01 16:31:56.958387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.958354 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2bcf\" (UniqueName: \"kubernetes.io/projected/7280418a-3eb2-48f4-813e-508c765be317-kube-api-access-k2bcf\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:56.958605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.958589 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7280418a-3eb2-48f4-813e-508c765be317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:56.958672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:56.958653 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7280418a-3eb2-48f4-813e-508c765be317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:57.060385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.060352 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7280418a-3eb2-48f4-813e-508c765be317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:57.060553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.060412 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7280418a-3eb2-48f4-813e-508c765be317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:57.060553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.060464 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2bcf\" (UniqueName: \"kubernetes.io/projected/7280418a-3eb2-48f4-813e-508c765be317-kube-api-access-k2bcf\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:57.061149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.061126 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7280418a-3eb2-48f4-813e-508c765be317-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:57.062895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.062874 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7280418a-3eb2-48f4-813e-508c765be317-maas-api-xks-ca\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:57.068090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.068071 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2bcf\" (UniqueName: \"kubernetes.io/projected/7280418a-3eb2-48f4-813e-508c765be317-kube-api-access-k2bcf\") pod \"maas-api-revoke-sub-66df19bb-ls59d\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:57.184846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.184810 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:57.469617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.469577 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:57.469968 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:31:57.469930 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7280418a_3eb2_48f4_813e_508c765be317.slice/crio-8a719d31321b8ec4ea19fe8b15326428e6abf123fea3fcec6d3492a87dec42c9 WatchSource:0}: Error finding container 8a719d31321b8ec4ea19fe8b15326428e6abf123fea3fcec6d3492a87dec42c9: Status 404 returned error can't find the container with id 8a719d31321b8ec4ea19fe8b15326428e6abf123fea3fcec6d3492a87dec42c9 Nov 01 16:31:57.470095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.470075 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d"] Nov 01 16:31:57.681801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:57.681769 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" event={"ID":"7280418a-3eb2-48f4-813e-508c765be317","Type":"ContainerStarted","Data":"8a719d31321b8ec4ea19fe8b15326428e6abf123fea3fcec6d3492a87dec42c9"} Nov 01 16:31:58.108997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:58.108964 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:58.205745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:58.205715 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:58.329652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:58.329614 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:58.431306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:58.431180 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:58.469636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:58.469606 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803fc24d-d984-49ee-a476-c6dbb72aa92e" path="/var/lib/kubelet/pods/803fc24d-d984-49ee-a476-c6dbb72aa92e/volumes" Nov 01 16:31:58.691324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:58.691227 4573 generic.go:356] "Generic (PLEG): container finished" podID="7280418a-3eb2-48f4-813e-508c765be317" containerID="10c8de49972c51f9f2815deae9d2c25c5bf45a303aae6e0b26f9a76cb952a1ca" exitCode=0 Nov 01 16:31:58.691486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:58.691316 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" event={"ID":"7280418a-3eb2-48f4-813e-508c765be317","Type":"ContainerDied","Data":"10c8de49972c51f9f2815deae9d2c25c5bf45a303aae6e0b26f9a76cb952a1ca"} Nov 01 16:31:58.836508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:58.836475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:59.525879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.525843 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:31:59.850808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.850783 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:31:59.923469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.923434 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7280418a-3eb2-48f4-813e-508c765be317-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7280418a-3eb2-48f4-813e-508c765be317-maas-api-xks-ca\") pod \"7280418a-3eb2-48f4-813e-508c765be317\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " Nov 01 16:31:59.923668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.923618 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7280418a-3eb2-48f4-813e-508c765be317-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7280418a-3eb2-48f4-813e-508c765be317-maas-api-openshift-ca\") pod \"7280418a-3eb2-48f4-813e-508c765be317\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " Nov 01 16:31:59.923668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.923651 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7280418a-3eb2-48f4-813e-508c765be317-kube-api-access-k2bcf\" (UniqueName: \"kubernetes.io/projected/7280418a-3eb2-48f4-813e-508c765be317-kube-api-access-k2bcf\") pod \"7280418a-3eb2-48f4-813e-508c765be317\" (UID: \"7280418a-3eb2-48f4-813e-508c765be317\") " Nov 01 16:31:59.924081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.924053 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7280418a-3eb2-48f4-813e-508c765be317-maas-api-openshift-ca" pod "7280418a-3eb2-48f4-813e-508c765be317" (UID: "7280418a-3eb2-48f4-813e-508c765be317"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:31:59.924190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.924175 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7280418a-3eb2-48f4-813e-508c765be317-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:31:59.925553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.925528 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7280418a-3eb2-48f4-813e-508c765be317-maas-api-xks-ca" pod "7280418a-3eb2-48f4-813e-508c765be317" (UID: "7280418a-3eb2-48f4-813e-508c765be317"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:31:59.925805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:31:59.925781 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7280418a-3eb2-48f4-813e-508c765be317-kube-api-access-k2bcf" pod "7280418a-3eb2-48f4-813e-508c765be317" (UID: "7280418a-3eb2-48f4-813e-508c765be317"). InnerVolumeSpecName "kube-api-access-k2bcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:00.025247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:00.025218 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7280418a-3eb2-48f4-813e-508c765be317-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:00.025247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:00.025243 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2bcf\" (UniqueName: \"kubernetes.io/projected/7280418a-3eb2-48f4-813e-508c765be317-kube-api-access-k2bcf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:00.260227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:00.260188 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:00.707596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:00.707558 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" event={"ID":"7280418a-3eb2-48f4-813e-508c765be317","Type":"ContainerDied","Data":"8a719d31321b8ec4ea19fe8b15326428e6abf123fea3fcec6d3492a87dec42c9"} Nov 01 16:32:00.707596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:00.707593 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d" Nov 01 16:32:00.708085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:00.707612 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a719d31321b8ec4ea19fe8b15326428e6abf123fea3fcec6d3492a87dec42c9" Nov 01 16:32:01.058586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:01.058501 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:01.684034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:01.684000 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:02.491984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:02.491950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:02.850809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:02.850697 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:02.953471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:02.953440 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:03.177642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.177605 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:03.286548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.286512 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:03.639163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.639125 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq"] Nov 01 16:32:03.639888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.639869 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7280418a-3eb2-48f4-813e-508c765be317" containerName="revoke-keys" Nov 01 16:32:03.639955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.639889 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7280418a-3eb2-48f4-813e-508c765be317" containerName="revoke-keys" Nov 01 16:32:03.640001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.639993 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7280418a-3eb2-48f4-813e-508c765be317" containerName="revoke-keys" Nov 01 16:32:03.643703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.643685 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.651131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.651106 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq"] Nov 01 16:32:03.677839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.677813 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:32:03.677989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.677908 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:32:03.791896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.791865 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.792081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.791908 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tngm\" (UniqueName: \"kubernetes.io/projected/84667fdb-a43f-4878-8ac7-e0a60bb32d06-kube-api-access-6tngm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.792081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.791997 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.812956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.812926 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j"] Nov 01 16:32:03.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.893711 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.893775 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tngm\" (UniqueName: \"kubernetes.io/projected/84667fdb-a43f-4878-8ac7-e0a60bb32d06-kube-api-access-6tngm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.893835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.893819 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.894500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.894478 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.896310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.896257 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.900998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.900965 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2"] Nov 01 16:32:03.902845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.902822 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tngm\" (UniqueName: \"kubernetes.io/projected/84667fdb-a43f-4878-8ac7-e0a60bb32d06-kube-api-access-6tngm\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.905829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.905810 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.908459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.908432 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:03.919399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.919377 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2"] Nov 01 16:32:03.963877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.963842 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:03.995119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995086 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.995260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995137 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.995260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995166 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.995388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995315 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-856jk\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-kube-api-access-856jk\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.995388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995343 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.995388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995366 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.995565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995532 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.995565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995561 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:03.995677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:03.995601 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.046662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.046595 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107156 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107223 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107259 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107352 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-856jk\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-kube-api-access-856jk\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107390 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107428 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107609 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107666 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.113541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.107720 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.124034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.123651 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.124937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.124182 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-certs\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.124937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.124468 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-data\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.127234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.127183 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-credential-socket\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.133635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.133599 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istiod-ca-cert\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.139696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.139640 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-podinfo\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.140999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.140962 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-856jk\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-kube-api-access-856jk\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.141361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.141332 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-envoy\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.142488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.142458 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-token\") pod \"e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.253944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.253700 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:04.253944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.253743 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq"] Nov 01 16:32:04.254416 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:04.254388 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84667fdb_a43f_4878_8ac7_e0a60bb32d06.slice/crio-c8cd5ea346bf9c3d16e8487869ee62ffddc49d7b5e2e9dcfebe882783366dc60 WatchSource:0}: Error finding container c8cd5ea346bf9c3d16e8487869ee62ffddc49d7b5e2e9dcfebe882783366dc60: Status 404 returned error can't find the container with id c8cd5ea346bf9c3d16e8487869ee62ffddc49d7b5e2e9dcfebe882783366dc60 Nov 01 16:32:04.545573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.545546 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2"] Nov 01 16:32:04.546026 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:04.546002 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2bc612_dcc3_4d1c_8aa6_24a0ba6d0b00.slice/crio-1767bfabc82297f6dff61de25716b43cd622bf5c750f3bb04fc6e70745e7176c WatchSource:0}: Error finding container 1767bfabc82297f6dff61de25716b43cd622bf5c750f3bb04fc6e70745e7176c: Status 404 returned error can't find the container with id 1767bfabc82297f6dff61de25716b43cd622bf5c750f3bb04fc6e70745e7176c Nov 01 16:32:04.548697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.548660 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:04.548798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.548734 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:04.548798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.548791 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:32:04.739914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.739873 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" event={"ID":"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00","Type":"ContainerStarted","Data":"281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1"} Nov 01 16:32:04.740361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.739941 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" event={"ID":"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00","Type":"ContainerStarted","Data":"1767bfabc82297f6dff61de25716b43cd622bf5c750f3bb04fc6e70745e7176c"} Nov 01 16:32:04.741673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.741646 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" event={"ID":"84667fdb-a43f-4878-8ac7-e0a60bb32d06","Type":"ContainerStarted","Data":"6eb08577ed4ba136fb6cf2d4fc5951501ee63a1c16dd5961d8e546d3922c2d39"} Nov 01 16:32:04.741832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.741690 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" event={"ID":"84667fdb-a43f-4878-8ac7-e0a60bb32d06","Type":"ContainerStarted","Data":"c8cd5ea346bf9c3d16e8487869ee62ffddc49d7b5e2e9dcfebe882783366dc60"} Nov 01 16:32:04.780869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.780821 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" podStartSLOduration=1.780806932 podStartE2EDuration="1.780806932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:04.765211751 +0000 UTC m=+2662.468377686" watchObservedRunningTime="2026-11-01 16:32:04.780806932 +0000 UTC m=+2662.483972867" Nov 01 16:32:04.782240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.782208 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" podStartSLOduration=1.782198253 podStartE2EDuration="1.782198253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:04.780194312 +0000 UTC m=+2662.483360250" watchObservedRunningTime="2026-11-01 16:32:04.782198253 +0000 UTC m=+2662.485364188" Nov 01 16:32:04.949447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:04.949412 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:05.254206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:05.254127 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:05.255947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:05.255926 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:32:05.256022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:05.255970 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 01 16:32:05.590873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:05.590550 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:05.751302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:05.751270 4573 generic.go:356] "Generic (PLEG): container finished" podID="84667fdb-a43f-4878-8ac7-e0a60bb32d06" containerID="6eb08577ed4ba136fb6cf2d4fc5951501ee63a1c16dd5961d8e546d3922c2d39" exitCode=0 Nov 01 16:32:05.751799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:05.751359 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" event={"ID":"84667fdb-a43f-4878-8ac7-e0a60bb32d06","Type":"ContainerDied","Data":"6eb08577ed4ba136fb6cf2d4fc5951501ee63a1c16dd5961d8e546d3922c2d39"} Nov 01 16:32:06.254400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:06.254366 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:32:06.254587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:06.254420 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 01 16:32:06.420829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:06.420793 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:06.988319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:06.988291 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:07.051141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.051040 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-xks-ca\") pod \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " Nov 01 16:32:07.051312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.051158 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84667fdb-a43f-4878-8ac7-e0a60bb32d06-kube-api-access-6tngm\" (UniqueName: \"kubernetes.io/projected/84667fdb-a43f-4878-8ac7-e0a60bb32d06-kube-api-access-6tngm\") pod \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " Nov 01 16:32:07.051312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.051225 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-openshift-ca\") pod \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\" (UID: \"84667fdb-a43f-4878-8ac7-e0a60bb32d06\") " Nov 01 16:32:07.052121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.052098 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-openshift-ca" pod "84667fdb-a43f-4878-8ac7-e0a60bb32d06" (UID: "84667fdb-a43f-4878-8ac7-e0a60bb32d06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:07.053483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.053459 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-xks-ca" pod "84667fdb-a43f-4878-8ac7-e0a60bb32d06" (UID: "84667fdb-a43f-4878-8ac7-e0a60bb32d06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:07.054430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.054409 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84667fdb-a43f-4878-8ac7-e0a60bb32d06-kube-api-access-6tngm" pod "84667fdb-a43f-4878-8ac7-e0a60bb32d06" (UID: "84667fdb-a43f-4878-8ac7-e0a60bb32d06"). InnerVolumeSpecName "kube-api-access-6tngm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:07.057416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.057356 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:07.153646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.153273 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:07.153646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.153311 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tngm\" (UniqueName: \"kubernetes.io/projected/84667fdb-a43f-4878-8ac7-e0a60bb32d06-kube-api-access-6tngm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:07.153646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.153329 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84667fdb-a43f-4878-8ac7-e0a60bb32d06-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:07.254176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.254143 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:32:07.254378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.254189 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 01 16:32:07.458371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.458333 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw"] Nov 01 16:32:07.459041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.459022 4573 cpu_manager.go:414] "containerMap: removing container" podUID="84667fdb-a43f-4878-8ac7-e0a60bb32d06" containerName="revoke-keys" Nov 01 16:32:07.459041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.459040 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84667fdb-a43f-4878-8ac7-e0a60bb32d06" containerName="revoke-keys" Nov 01 16:32:07.459235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.459169 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="84667fdb-a43f-4878-8ac7-e0a60bb32d06" containerName="revoke-keys" Nov 01 16:32:07.481770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.471017 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.497081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.496918 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw"] Nov 01 16:32:07.560351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.560319 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.560679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.560663 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.560783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.560772 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm9gl\" (UniqueName: \"kubernetes.io/projected/7a3ab6bb-2808-417d-997e-6b3365904650-kube-api-access-dm9gl\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.661684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.661647 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.661912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.661728 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.661912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.661768 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm9gl\" (UniqueName: \"kubernetes.io/projected/7a3ab6bb-2808-417d-997e-6b3365904650-kube-api-access-dm9gl\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.677779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.662490 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.677779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.674856 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.688629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.688576 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm9gl\" (UniqueName: \"kubernetes.io/projected/7a3ab6bb-2808-417d-997e-6b3365904650-kube-api-access-dm9gl\") pod \"maas-api-revoke-sub-7bff2f0d-4l2rw\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:07.738687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.738611 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:07.792770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.792165 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" event={"ID":"84667fdb-a43f-4878-8ac7-e0a60bb32d06","Type":"ContainerDied","Data":"c8cd5ea346bf9c3d16e8487869ee62ffddc49d7b5e2e9dcfebe882783366dc60"} Nov 01 16:32:07.792770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.792213 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8cd5ea346bf9c3d16e8487869ee62ffddc49d7b5e2e9dcfebe882783366dc60" Nov 01 16:32:07.792770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.792261 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq" Nov 01 16:32:07.813016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:07.807523 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:08.126851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:08.126814 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw"] Nov 01 16:32:08.127294 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:08.127272 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a3ab6bb_2808_417d_997e_6b3365904650.slice/crio-735b1b9c9444470fc94546051720ae2ce6508deffc6d699d119f42942d785917 WatchSource:0}: Error finding container 735b1b9c9444470fc94546051720ae2ce6508deffc6d699d119f42942d785917: Status 404 returned error can't find the container with id 735b1b9c9444470fc94546051720ae2ce6508deffc6d699d119f42942d785917 Nov 01 16:32:08.254507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:08.254369 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:32:08.254507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:08.254418 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 01 16:32:08.387617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:08.387327 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:08.800887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:08.800847 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" event={"ID":"7a3ab6bb-2808-417d-997e-6b3365904650","Type":"ContainerStarted","Data":"fb011cb7cdc383a24b61292b0b534a776c94801de0e68e5d6ddbb8b107eefaec"} Nov 01 16:32:08.801192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:08.801168 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" event={"ID":"7a3ab6bb-2808-417d-997e-6b3365904650","Type":"ContainerStarted","Data":"735b1b9c9444470fc94546051720ae2ce6508deffc6d699d119f42942d785917"} Nov 01 16:32:08.817317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:08.817275 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" podStartSLOduration=1.817262748 podStartE2EDuration="1.817262748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:08.815352689 +0000 UTC m=+2666.518518626" watchObservedRunningTime="2026-11-01 16:32:08.817262748 +0000 UTC m=+2666.520428684" Nov 01 16:32:09.148560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:09.148525 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs"] Nov 01 16:32:09.190791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:09.184495 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:09.254516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:09.254488 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:32:09.254732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:09.254711 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 01 16:32:09.294137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:09.270151 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:09.810539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:09.810506 4573 generic.go:356] "Generic (PLEG): container finished" podID="7a3ab6bb-2808-417d-997e-6b3365904650" containerID="fb011cb7cdc383a24b61292b0b534a776c94801de0e68e5d6ddbb8b107eefaec" exitCode=0 Nov 01 16:32:09.810795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:09.810552 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" event={"ID":"7a3ab6bb-2808-417d-997e-6b3365904650","Type":"ContainerDied","Data":"fb011cb7cdc383a24b61292b0b534a776c94801de0e68e5d6ddbb8b107eefaec"} Nov 01 16:32:09.987650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:09.987521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:10.254392 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:10.254359 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" start-of-body= Nov 01 16:32:10.254934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:10.254516 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.47:15021/healthz/ready\": dial tcp 10.133.1.47:15021: connect: connection refused" Nov 01 16:32:10.665501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:10.665463 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:11.008206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.008179 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:11.046114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.046080 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-xks-ca\") pod \"7a3ab6bb-2808-417d-997e-6b3365904650\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " Nov 01 16:32:11.046305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.046169 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-openshift-ca\") pod \"7a3ab6bb-2808-417d-997e-6b3365904650\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " Nov 01 16:32:11.047173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.047133 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-openshift-ca" pod "7a3ab6bb-2808-417d-997e-6b3365904650" (UID: "7a3ab6bb-2808-417d-997e-6b3365904650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:11.049745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.049711 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-xks-ca" pod "7a3ab6bb-2808-417d-997e-6b3365904650" (UID: "7a3ab6bb-2808-417d-997e-6b3365904650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:11.147560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.147513 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a3ab6bb-2808-417d-997e-6b3365904650-kube-api-access-dm9gl\" (UniqueName: \"kubernetes.io/projected/7a3ab6bb-2808-417d-997e-6b3365904650-kube-api-access-dm9gl\") pod \"7a3ab6bb-2808-417d-997e-6b3365904650\" (UID: \"7a3ab6bb-2808-417d-997e-6b3365904650\") " Nov 01 16:32:11.148224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.148201 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:11.148224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.148224 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a3ab6bb-2808-417d-997e-6b3365904650-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:11.149859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.149835 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a3ab6bb-2808-417d-997e-6b3365904650-kube-api-access-dm9gl" pod "7a3ab6bb-2808-417d-997e-6b3365904650" (UID: "7a3ab6bb-2808-417d-997e-6b3365904650"). InnerVolumeSpecName "kube-api-access-dm9gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:11.249296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.249216 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm9gl\" (UniqueName: \"kubernetes.io/projected/7a3ab6bb-2808-417d-997e-6b3365904650-kube-api-access-dm9gl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:11.259072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.259045 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:11.259519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.259485 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:11.260161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.260142 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:32:11.615125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.615093 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:11.830560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.830532 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" Nov 01 16:32:11.830775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.830531 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw" event={"ID":"7a3ab6bb-2808-417d-997e-6b3365904650","Type":"ContainerDied","Data":"735b1b9c9444470fc94546051720ae2ce6508deffc6d699d119f42942d785917"} Nov 01 16:32:11.830775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:11.830682 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="735b1b9c9444470fc94546051720ae2ce6508deffc6d699d119f42942d785917" Nov 01 16:32:12.248394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:12.248365 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:13.035218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:13.035183 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:13.687450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:13.687415 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:13.835189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:13.835149 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:13.940576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:13.940485 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:14.153168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:14.153110 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" podUID="31dd6eb6-7339-4276-8332-e89ec52d1d5e" containerName="payload-processing" containerID="cri-o://2993fc65690e1d5baa0516df25ea6ea4b50c2ded10775ddeb41e3be76fdbb468" gracePeriod=25 Nov 01 16:32:14.477916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:14.477880 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:15.150474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:15.150443 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:15.775534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:15.775471 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:16.316705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.316663 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg"] Nov 01 16:32:16.317383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.317368 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7a3ab6bb-2808-417d-997e-6b3365904650" containerName="revoke-keys" Nov 01 16:32:16.317434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.317384 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a3ab6bb-2808-417d-997e-6b3365904650" containerName="revoke-keys" Nov 01 16:32:16.317493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.317484 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a3ab6bb-2808-417d-997e-6b3365904650" containerName="revoke-keys" Nov 01 16:32:16.321301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.321278 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.325523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.325495 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:32:16.325665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.325576 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:32:16.332166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.332142 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg"] Nov 01 16:32:16.356793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.356728 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs"] Nov 01 16:32:16.356997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.356951 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" podUID="a1b57bd1-e85b-46bb-a367-d921b85a7201" containerName="maas-api" containerID="cri-o://5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899" gracePeriod=30 Nov 01 16:32:16.389421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.389372 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:16.421402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.421361 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.421600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.421527 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.421738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.421707 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8w8t\" (UniqueName: \"kubernetes.io/projected/f8f57b93-a805-400f-b65f-f0e0140a3e31-kube-api-access-d8w8t\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.433348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.433295 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq"] Nov 01 16:32:16.433937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.433574 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" podUID="eba74750-f945-4e87-8d55-221be97e33a4" containerName="payload-pre-processing" containerID="cri-o://aceb1e7f814946480be1b6919c80b1a58c92e877e23003d6081a3690ae4b133b" gracePeriod=30 Nov 01 16:32:16.479899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.479870 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:16.522943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.522907 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.523140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.522977 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.523140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.523031 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8w8t\" (UniqueName: \"kubernetes.io/projected/f8f57b93-a805-400f-b65f-f0e0140a3e31-kube-api-access-d8w8t\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.523817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.523793 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.525380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.525357 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.531679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.531656 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8w8t\" (UniqueName: \"kubernetes.io/projected/f8f57b93-a805-400f-b65f-f0e0140a3e31-kube-api-access-d8w8t\") pod \"maas-api-revoke-sub-061ca36f-hmgqg\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.585317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.585143 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:16.659274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.659244 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:16.676385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.675536 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:16.873044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.872822 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:32:16.890443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.890371 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:16.929412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.929379 4573 generic.go:356] "Generic (PLEG): container finished" podID="a1b57bd1-e85b-46bb-a367-d921b85a7201" containerID="5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899" exitCode=0 Nov 01 16:32:16.929552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.929415 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" Nov 01 16:32:16.929552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.929514 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" event={"ID":"a1b57bd1-e85b-46bb-a367-d921b85a7201","Type":"ContainerDied","Data":"5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899"} Nov 01 16:32:16.929635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.929564 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs" event={"ID":"a1b57bd1-e85b-46bb-a367-d921b85a7201","Type":"ContainerDied","Data":"dd01344dd4c9addc8bcb5b774554f4b143b75cfa041d11ef4c365bc852685fe1"} Nov 01 16:32:16.929635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.929596 4573 scope.go:122] "RemoveContainer" containerID="5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899" Nov 01 16:32:16.932242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.932223 4573 generic.go:356] "Generic (PLEG): container finished" podID="eba74750-f945-4e87-8d55-221be97e33a4" containerID="aceb1e7f814946480be1b6919c80b1a58c92e877e23003d6081a3690ae4b133b" exitCode=0 Nov 01 16:32:16.932407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.932395 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" event={"ID":"eba74750-f945-4e87-8d55-221be97e33a4","Type":"ContainerDied","Data":"aceb1e7f814946480be1b6919c80b1a58c92e877e23003d6081a3690ae4b133b"} Nov 01 16:32:16.935299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.935036 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:32:16.942481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.942457 4573 scope.go:122] "RemoveContainer" containerID="5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899" Nov 01 16:32:16.942822 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:32:16.942796 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899\": container with ID starting with 5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899 not found: ID does not exist" containerID="5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899" Nov 01 16:32:16.942916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:16.942834 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899"} err="failed to get container status \"5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899\": rpc error: code = NotFound desc = could not find container \"5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899\": container with ID starting with 5fadea005ead68f357d2d05d2947c4ab1748e4822fc83f8eebe2b265a7871899 not found: ID does not exist" Nov 01 16:32:17.002176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.002139 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-maas-api-tls\") pod \"a1b57bd1-e85b-46bb-a367-d921b85a7201\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " Nov 01 16:32:17.002358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.002212 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b57bd1-e85b-46bb-a367-d921b85a7201-kube-api-access-vmbx4\" (UniqueName: \"kubernetes.io/projected/a1b57bd1-e85b-46bb-a367-d921b85a7201-kube-api-access-vmbx4\") pod \"a1b57bd1-e85b-46bb-a367-d921b85a7201\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " Nov 01 16:32:17.002358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.002329 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eba74750-f945-4e87-8d55-221be97e33a4-kube-api-access-5f64h\" (UniqueName: \"kubernetes.io/projected/eba74750-f945-4e87-8d55-221be97e33a4-kube-api-access-5f64h\") pod \"eba74750-f945-4e87-8d55-221be97e33a4\" (UID: \"eba74750-f945-4e87-8d55-221be97e33a4\") " Nov 01 16:32:17.002674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.002651 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eba74750-f945-4e87-8d55-221be97e33a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba74750-f945-4e87-8d55-221be97e33a4-plugins-config-volume\") pod \"eba74750-f945-4e87-8d55-221be97e33a4\" (UID: \"eba74750-f945-4e87-8d55-221be97e33a4\") " Nov 01 16:32:17.003032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.003010 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b57bd1-e85b-46bb-a367-d921b85a7201-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b57bd1-e85b-46bb-a367-d921b85a7201-openshift-service-ca\") pod \"a1b57bd1-e85b-46bb-a367-d921b85a7201\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " Nov 01 16:32:17.003180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.003125 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-metrics-certs\") pod \"a1b57bd1-e85b-46bb-a367-d921b85a7201\" (UID: \"a1b57bd1-e85b-46bb-a367-d921b85a7201\") " Nov 01 16:32:17.003316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.003147 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba74750-f945-4e87-8d55-221be97e33a4-plugins-config-volume" pod "eba74750-f945-4e87-8d55-221be97e33a4" (UID: "eba74750-f945-4e87-8d55-221be97e33a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:17.005870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.003335 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b57bd1-e85b-46bb-a367-d921b85a7201-openshift-service-ca" pod "a1b57bd1-e85b-46bb-a367-d921b85a7201" (UID: "a1b57bd1-e85b-46bb-a367-d921b85a7201"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:17.005870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.003789 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1b57bd1-e85b-46bb-a367-d921b85a7201-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:17.005870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.003810 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eba74750-f945-4e87-8d55-221be97e33a4-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:17.005870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.004596 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b57bd1-e85b-46bb-a367-d921b85a7201-kube-api-access-vmbx4" pod "a1b57bd1-e85b-46bb-a367-d921b85a7201" (UID: "a1b57bd1-e85b-46bb-a367-d921b85a7201"). InnerVolumeSpecName "kube-api-access-vmbx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:17.005870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.004908 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eba74750-f945-4e87-8d55-221be97e33a4-kube-api-access-5f64h" pod "eba74750-f945-4e87-8d55-221be97e33a4" (UID: "eba74750-f945-4e87-8d55-221be97e33a4"). InnerVolumeSpecName "kube-api-access-5f64h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:17.005870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.005129 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-maas-api-tls" pod "a1b57bd1-e85b-46bb-a367-d921b85a7201" (UID: "a1b57bd1-e85b-46bb-a367-d921b85a7201"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:17.005870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.005398 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-metrics-certs" pod "a1b57bd1-e85b-46bb-a367-d921b85a7201" (UID: "a1b57bd1-e85b-46bb-a367-d921b85a7201"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:17.093892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.093856 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:17.104813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.104786 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg"] Nov 01 16:32:17.105548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.105520 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:17.105659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.105555 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a1b57bd1-e85b-46bb-a367-d921b85a7201-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:17.105659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.105571 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmbx4\" (UniqueName: \"kubernetes.io/projected/a1b57bd1-e85b-46bb-a367-d921b85a7201-kube-api-access-vmbx4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:17.105659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.105590 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f64h\" (UniqueName: \"kubernetes.io/projected/eba74750-f945-4e87-8d55-221be97e33a4-kube-api-access-5f64h\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:17.105952 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:17.105928 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f57b93_a805_400f_b65f_f0e0140a3e31.slice/crio-bc2faf07ca636e29dc0232e05ee21f9784cf049df4241d3485d686ab927df096 WatchSource:0}: Error finding container bc2faf07ca636e29dc0232e05ee21f9784cf049df4241d3485d686ab927df096: Status 404 returned error can't find the container with id bc2faf07ca636e29dc0232e05ee21f9784cf049df4241d3485d686ab927df096 Nov 01 16:32:17.254880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.254850 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs"] Nov 01 16:32:17.258625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.258597 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-028c46-5c7f6799cc-kwzzs"] Nov 01 16:32:17.669024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.668937 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:17.896554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.896514 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz"] Nov 01 16:32:17.897214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.897198 4573 cpu_manager.go:414] "containerMap: removing container" podUID="eba74750-f945-4e87-8d55-221be97e33a4" containerName="payload-pre-processing" Nov 01 16:32:17.897266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.897215 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eba74750-f945-4e87-8d55-221be97e33a4" containerName="payload-pre-processing" Nov 01 16:32:17.897304 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.897267 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a1b57bd1-e85b-46bb-a367-d921b85a7201" containerName="maas-api" Nov 01 16:32:17.897304 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.897273 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b57bd1-e85b-46bb-a367-d921b85a7201" containerName="maas-api" Nov 01 16:32:17.897386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.897375 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b57bd1-e85b-46bb-a367-d921b85a7201" containerName="maas-api" Nov 01 16:32:17.897422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.897395 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="eba74750-f945-4e87-8d55-221be97e33a4" containerName="payload-pre-processing" Nov 01 16:32:17.901027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.901009 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:17.907736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.907710 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz"] Nov 01 16:32:17.944691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.944606 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" event={"ID":"f8f57b93-a805-400f-b65f-f0e0140a3e31","Type":"ContainerStarted","Data":"461aaf765d7418c3e77c5c41c6182f1d29456a86a973966f9a196f34d75f9797"} Nov 01 16:32:17.944691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.944661 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" event={"ID":"f8f57b93-a805-400f-b65f-f0e0140a3e31","Type":"ContainerStarted","Data":"bc2faf07ca636e29dc0232e05ee21f9784cf049df4241d3485d686ab927df096"} Nov 01 16:32:17.947704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.947676 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" event={"ID":"eba74750-f945-4e87-8d55-221be97e33a4","Type":"ContainerDied","Data":"28a42c5ebf9d4c0c72057ac41282b0b07ba2359e091a2fdc25b8324df2b36499"} Nov 01 16:32:17.947884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.947705 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq" Nov 01 16:32:17.947884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.947739 4573 scope.go:122] "RemoveContainer" containerID="aceb1e7f814946480be1b6919c80b1a58c92e877e23003d6081a3690ae4b133b" Nov 01 16:32:17.965746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.965705 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" podStartSLOduration=1.965687637 podStartE2EDuration="1.965687637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:17.960477401 +0000 UTC m=+2675.663643337" watchObservedRunningTime="2026-11-01 16:32:17.965687637 +0000 UTC m=+2675.668853573" Nov 01 16:32:17.977866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.977844 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq"] Nov 01 16:32:17.982797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:17.982776 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-028c46-769655699f-kzrnq"] Nov 01 16:32:18.020595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.020556 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d9dd2-b607-43af-a447-1650accc744f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.020845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.020812 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d9dd2-b607-43af-a447-1650accc744f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.020942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.020920 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbm4\" (UniqueName: \"kubernetes.io/projected/791d9dd2-b607-43af-a447-1650accc744f-kube-api-access-hgbm4\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.122219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.122184 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d9dd2-b607-43af-a447-1650accc744f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.122367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.122291 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbm4\" (UniqueName: \"kubernetes.io/projected/791d9dd2-b607-43af-a447-1650accc744f-kube-api-access-hgbm4\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.122440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.122405 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d9dd2-b607-43af-a447-1650accc744f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.122850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.122825 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d9dd2-b607-43af-a447-1650accc744f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.124517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.124498 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d9dd2-b607-43af-a447-1650accc744f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.136874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.136853 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbm4\" (UniqueName: \"kubernetes.io/projected/791d9dd2-b607-43af-a447-1650accc744f-kube-api-access-hgbm4\") pod \"maas-api-revoke-sub-3abcfa78-wh7dz\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.219438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.219365 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:18.459612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.459577 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b57bd1-e85b-46bb-a367-d921b85a7201" path="/var/lib/kubelet/pods/a1b57bd1-e85b-46bb-a367-d921b85a7201/volumes" Nov 01 16:32:18.460130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.460107 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eba74750-f945-4e87-8d55-221be97e33a4" path="/var/lib/kubelet/pods/eba74750-f945-4e87-8d55-221be97e33a4/volumes" Nov 01 16:32:18.488587 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.488513 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:32:18.513284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.513258 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz"] Nov 01 16:32:18.513420 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:18.513261 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791d9dd2_b607_43af_a447_1650accc744f.slice/crio-cc5dd5b37c7e57f2f7e9c557210a949c2838c48da2162394ec06d6f26c5f1fa8 WatchSource:0}: Error finding container cc5dd5b37c7e57f2f7e9c557210a949c2838c48da2162394ec06d6f26c5f1fa8: Status 404 returned error can't find the container with id cc5dd5b37c7e57f2f7e9c557210a949c2838c48da2162394ec06d6f26c5f1fa8 Nov 01 16:32:18.584023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.583997 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:18.835796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.835704 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" containerID="cri-o://76c42123c400cb65c94a7bf4bf7a6b564b12a2244af65733f185c4dd37cef467" gracePeriod=45 Nov 01 16:32:18.957301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.957270 4573 generic.go:356] "Generic (PLEG): container finished" podID="f8f57b93-a805-400f-b65f-f0e0140a3e31" containerID="461aaf765d7418c3e77c5c41c6182f1d29456a86a973966f9a196f34d75f9797" exitCode=0 Nov 01 16:32:18.957647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.957354 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" event={"ID":"f8f57b93-a805-400f-b65f-f0e0140a3e31","Type":"ContainerDied","Data":"461aaf765d7418c3e77c5c41c6182f1d29456a86a973966f9a196f34d75f9797"} Nov 01 16:32:18.959965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.959940 4573 generic.go:356] "Generic (PLEG): container finished" podID="813003eb-88ca-4744-919e-085b1ce97da7" containerID="76c42123c400cb65c94a7bf4bf7a6b564b12a2244af65733f185c4dd37cef467" exitCode=0 Nov 01 16:32:18.960105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.960011 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" event={"ID":"813003eb-88ca-4744-919e-085b1ce97da7","Type":"ContainerDied","Data":"76c42123c400cb65c94a7bf4bf7a6b564b12a2244af65733f185c4dd37cef467"} Nov 01 16:32:18.961610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.961581 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" event={"ID":"791d9dd2-b607-43af-a447-1650accc744f","Type":"ContainerStarted","Data":"f76f116203928cde5027a28e72e1435fbe5c3e0c2507a4a577a8ce3d29b93e56"} Nov 01 16:32:18.961718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.961632 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" event={"ID":"791d9dd2-b607-43af-a447-1650accc744f","Type":"ContainerStarted","Data":"cc5dd5b37c7e57f2f7e9c557210a949c2838c48da2162394ec06d6f26c5f1fa8"} Nov 01 16:32:18.992118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:18.992054 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" podStartSLOduration=1.992015479 podStartE2EDuration="1.992015479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:18.991321133 +0000 UTC m=+2676.694487069" watchObservedRunningTime="2026-11-01 16:32:18.992015479 +0000 UTC m=+2676.695181416" Nov 01 16:32:19.034147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.034035 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws"] Nov 01 16:32:19.039693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.039655 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eb3fe9dd-t9cws"] Nov 01 16:32:19.112666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.112636 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:32:19.244741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.244703 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/813003eb-88ca-4744-919e-085b1ce97da7-kube-api-access-lqlpk\" (UniqueName: \"kubernetes.io/projected/813003eb-88ca-4744-919e-085b1ce97da7-kube-api-access-lqlpk\") pod \"813003eb-88ca-4744-919e-085b1ce97da7\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " Nov 01 16:32:19.244948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.244776 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/813003eb-88ca-4744-919e-085b1ce97da7-tls-certs\" (UniqueName: \"kubernetes.io/secret/813003eb-88ca-4744-919e-085b1ce97da7-tls-certs\") pod \"813003eb-88ca-4744-919e-085b1ce97da7\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " Nov 01 16:32:19.245046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.245027 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-home\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-home\") pod \"813003eb-88ca-4744-919e-085b1ce97da7\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " Nov 01 16:32:19.245115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.245089 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-tmp-dir\") pod \"813003eb-88ca-4744-919e-085b1ce97da7\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " Nov 01 16:32:19.245183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.245119 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-model-cache\") pod \"813003eb-88ca-4744-919e-085b1ce97da7\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " Nov 01 16:32:19.245183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.245160 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-dshm\") pod \"813003eb-88ca-4744-919e-085b1ce97da7\" (UID: \"813003eb-88ca-4744-919e-085b1ce97da7\") " Nov 01 16:32:19.245447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.245407 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-tmp-dir" pod "813003eb-88ca-4744-919e-085b1ce97da7" (UID: "813003eb-88ca-4744-919e-085b1ce97da7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.245601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.245584 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-home" pod "813003eb-88ca-4744-919e-085b1ce97da7" (UID: "813003eb-88ca-4744-919e-085b1ce97da7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.245820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.245799 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-model-cache" pod "813003eb-88ca-4744-919e-085b1ce97da7" (UID: "813003eb-88ca-4744-919e-085b1ce97da7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.246260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.246234 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.246423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.246263 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.246423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.246280 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.247994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.247951 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/813003eb-88ca-4744-919e-085b1ce97da7-kube-api-access-lqlpk" pod "813003eb-88ca-4744-919e-085b1ce97da7" (UID: "813003eb-88ca-4744-919e-085b1ce97da7"). InnerVolumeSpecName "kube-api-access-lqlpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:19.248216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.248189 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-dshm" pod "813003eb-88ca-4744-919e-085b1ce97da7" (UID: "813003eb-88ca-4744-919e-085b1ce97da7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:19.248565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.248543 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/813003eb-88ca-4744-919e-085b1ce97da7-tls-certs" pod "813003eb-88ca-4744-919e-085b1ce97da7" (UID: "813003eb-88ca-4744-919e-085b1ce97da7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:19.347601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.347562 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqlpk\" (UniqueName: \"kubernetes.io/projected/813003eb-88ca-4744-919e-085b1ce97da7-kube-api-access-lqlpk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.347601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.347593 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/813003eb-88ca-4744-919e-085b1ce97da7-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.347601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.347605 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/813003eb-88ca-4744-919e-085b1ce97da7-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:19.615742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:19.615706 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:20.052838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.038373 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" event={"ID":"813003eb-88ca-4744-919e-085b1ce97da7","Type":"ContainerDied","Data":"f389ffef76db0b90b5e0fc86b5f1886bf3c141b0a4059b40429663a547e9326a"} Nov 01 16:32:20.052838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.038440 4573 scope.go:122] "RemoveContainer" containerID="76c42123c400cb65c94a7bf4bf7a6b564b12a2244af65733f185c4dd37cef467" Nov 01 16:32:20.052838 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.038583 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j" Nov 01 16:32:20.142954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.118313 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j"] Nov 01 16:32:20.143736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.127776 4573 generic.go:356] "Generic (PLEG): container finished" podID="791d9dd2-b607-43af-a447-1650accc744f" containerID="f76f116203928cde5027a28e72e1435fbe5c3e0c2507a4a577a8ce3d29b93e56" exitCode=0 Nov 01 16:32:20.166440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.166402 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" event={"ID":"791d9dd2-b607-43af-a447-1650accc744f","Type":"ContainerDied","Data":"f76f116203928cde5027a28e72e1435fbe5c3e0c2507a4a577a8ce3d29b93e56"} Nov 01 16:32:20.166634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.166612 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-43fac8/distinct-w6-43fac8-kserve-6bd5b9775b-vnr2j"] Nov 01 16:32:20.314053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.313696 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" podUID="31dd6eb6-7339-4276-8332-e89ec52d1d5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:20.438779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.438726 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:20.460015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.459986 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5367ac19-8e88-4933-9ffc-52ae78419ac2" path="/var/lib/kubelet/pods/5367ac19-8e88-4933-9ffc-52ae78419ac2/volumes" Nov 01 16:32:20.460635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.460613 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="813003eb-88ca-4744-919e-085b1ce97da7" path="/var/lib/kubelet/pods/813003eb-88ca-4744-919e-085b1ce97da7/volumes" Nov 01 16:32:20.467678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.467648 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:20.535061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.532148 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-openshift-ca" pod "f8f57b93-a805-400f-b65f-f0e0140a3e31" (UID: "f8f57b93-a805-400f-b65f-f0e0140a3e31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:20.535061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.532259 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-openshift-ca\") pod \"f8f57b93-a805-400f-b65f-f0e0140a3e31\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " Nov 01 16:32:20.535061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.532820 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f57b93-a805-400f-b65f-f0e0140a3e31-kube-api-access-d8w8t\" (UniqueName: \"kubernetes.io/projected/f8f57b93-a805-400f-b65f-f0e0140a3e31-kube-api-access-d8w8t\") pod \"f8f57b93-a805-400f-b65f-f0e0140a3e31\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " Nov 01 16:32:20.535798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.535697 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-xks-ca\") pod \"f8f57b93-a805-400f-b65f-f0e0140a3e31\" (UID: \"f8f57b93-a805-400f-b65f-f0e0140a3e31\") " Nov 01 16:32:20.546247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.536827 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:20.547178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.547145 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f57b93-a805-400f-b65f-f0e0140a3e31-kube-api-access-d8w8t" pod "f8f57b93-a805-400f-b65f-f0e0140a3e31" (UID: "f8f57b93-a805-400f-b65f-f0e0140a3e31"). InnerVolumeSpecName "kube-api-access-d8w8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:20.550338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.550306 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-xks-ca" pod "f8f57b93-a805-400f-b65f-f0e0140a3e31" (UID: "f8f57b93-a805-400f-b65f-f0e0140a3e31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:20.638394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.638359 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8w8t\" (UniqueName: \"kubernetes.io/projected/f8f57b93-a805-400f-b65f-f0e0140a3e31-kube-api-access-d8w8t\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:20.638394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:20.638388 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f57b93-a805-400f-b65f-f0e0140a3e31-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:21.151907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.151875 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" Nov 01 16:32:21.152408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.151915 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg" event={"ID":"f8f57b93-a805-400f-b65f-f0e0140a3e31","Type":"ContainerDied","Data":"bc2faf07ca636e29dc0232e05ee21f9784cf049df4241d3485d686ab927df096"} Nov 01 16:32:21.152408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.151965 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc2faf07ca636e29dc0232e05ee21f9784cf049df4241d3485d686ab927df096" Nov 01 16:32:21.162415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.162389 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:21.343987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.343961 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:21.380299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.380264 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/791d9dd2-b607-43af-a447-1650accc744f-kube-api-access-hgbm4\" (UniqueName: \"kubernetes.io/projected/791d9dd2-b607-43af-a447-1650accc744f-kube-api-access-hgbm4\") pod \"791d9dd2-b607-43af-a447-1650accc744f\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " Nov 01 16:32:21.380515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.380493 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/791d9dd2-b607-43af-a447-1650accc744f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d9dd2-b607-43af-a447-1650accc744f-maas-api-openshift-ca\") pod \"791d9dd2-b607-43af-a447-1650accc744f\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " Nov 01 16:32:21.380572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.380542 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/791d9dd2-b607-43af-a447-1650accc744f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d9dd2-b607-43af-a447-1650accc744f-maas-api-xks-ca\") pod \"791d9dd2-b607-43af-a447-1650accc744f\" (UID: \"791d9dd2-b607-43af-a447-1650accc744f\") " Nov 01 16:32:21.380903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.380872 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/791d9dd2-b607-43af-a447-1650accc744f-maas-api-openshift-ca" pod "791d9dd2-b607-43af-a447-1650accc744f" (UID: "791d9dd2-b607-43af-a447-1650accc744f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:21.381196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.381172 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/791d9dd2-b607-43af-a447-1650accc744f-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:21.382651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.382631 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791d9dd2-b607-43af-a447-1650accc744f-kube-api-access-hgbm4" pod "791d9dd2-b607-43af-a447-1650accc744f" (UID: "791d9dd2-b607-43af-a447-1650accc744f"). InnerVolumeSpecName "kube-api-access-hgbm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:21.383180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.383163 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791d9dd2-b607-43af-a447-1650accc744f-maas-api-xks-ca" pod "791d9dd2-b607-43af-a447-1650accc744f" (UID: "791d9dd2-b607-43af-a447-1650accc744f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:21.482202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.482172 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/791d9dd2-b607-43af-a447-1650accc744f-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:21.482202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.482203 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbm4\" (UniqueName: \"kubernetes.io/projected/791d9dd2-b607-43af-a447-1650accc744f-kube-api-access-hgbm4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:21.966188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:21.966139 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:22.161664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:22.161637 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" Nov 01 16:32:22.162243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:22.161636 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz" event={"ID":"791d9dd2-b607-43af-a447-1650accc744f","Type":"ContainerDied","Data":"cc5dd5b37c7e57f2f7e9c557210a949c2838c48da2162394ec06d6f26c5f1fa8"} Nov 01 16:32:22.162243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:22.161795 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc5dd5b37c7e57f2f7e9c557210a949c2838c48da2162394ec06d6f26c5f1fa8" Nov 01 16:32:22.754092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:22.754052 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:23.323526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:23.323494 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:23.406460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:23.406427 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:23.483123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:23.483088 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:23.633872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:23.633820 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r"] Nov 01 16:32:23.634378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:23.634341 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" containerID="cri-o://c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b" gracePeriod=30 Nov 01 16:32:23.663912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:23.663884 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:24.024844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.024733 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq"] Nov 01 16:32:24.028493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.028465 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c0c629a-pg9vq"] Nov 01 16:32:24.090629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.090599 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524"] Nov 01 16:32:24.091254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091240 4573 cpu_manager.go:414] "containerMap: removing container" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" Nov 01 16:32:24.091324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091255 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" Nov 01 16:32:24.091324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091277 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f8f57b93-a805-400f-b65f-f0e0140a3e31" containerName="revoke-keys" Nov 01 16:32:24.091324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091282 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f57b93-a805-400f-b65f-f0e0140a3e31" containerName="revoke-keys" Nov 01 16:32:24.091324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091298 4573 cpu_manager.go:414] "containerMap: removing container" podUID="791d9dd2-b607-43af-a447-1650accc744f" containerName="revoke-keys" Nov 01 16:32:24.091324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091303 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="791d9dd2-b607-43af-a447-1650accc744f" containerName="revoke-keys" Nov 01 16:32:24.091497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091385 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="813003eb-88ca-4744-919e-085b1ce97da7" containerName="main" Nov 01 16:32:24.091497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091396 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f57b93-a805-400f-b65f-f0e0140a3e31" containerName="revoke-keys" Nov 01 16:32:24.091497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.091403 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="791d9dd2-b607-43af-a447-1650accc744f" containerName="revoke-keys" Nov 01 16:32:24.095121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.095100 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.102693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.102666 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524"] Nov 01 16:32:24.120144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.120120 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:32:24.120280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.120254 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:32:24.144094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.144056 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.144259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.144245 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.144306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.144295 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbvbl\" (UniqueName: \"kubernetes.io/projected/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-kube-api-access-xbvbl\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.246009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.245973 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbvbl\" (UniqueName: \"kubernetes.io/projected/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-kube-api-access-xbvbl\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.246221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.246095 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.246298 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.246239 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.246812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.246745 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.248626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.248602 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.253927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.253905 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbvbl\" (UniqueName: \"kubernetes.io/projected/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-kube-api-access-xbvbl\") pod \"maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.364323 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:32:24.364289 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:32:24.413514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.413480 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:24.449454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.449422 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:24.460974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.460948 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a206da-61dd-410b-ba81-63aa82a1630a" path="/var/lib/kubelet/pods/94a206da-61dd-410b-ba81-63aa82a1630a/volumes" Nov 01 16:32:24.697646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:24.697617 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524"] Nov 01 16:32:24.698324 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:24.698290 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2fdfa4_7277_41fe_bbf9_97f81116ad25.slice/crio-b3411aa81030749fb5d06899ca979a87b1dd6659072efc346a25f3f1e2020832 WatchSource:0}: Error finding container b3411aa81030749fb5d06899ca979a87b1dd6659072efc346a25f3f1e2020832: Status 404 returned error can't find the container with id b3411aa81030749fb5d06899ca979a87b1dd6659072efc346a25f3f1e2020832 Nov 01 16:32:25.072871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.072836 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:25.190944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.190906 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" event={"ID":"4f2fdfa4-7277-41fe-bbf9-97f81116ad25","Type":"ContainerStarted","Data":"a470048b500f723a50a8a48e74d78a8fa97dace43d1f27c4021904076931c02e"} Nov 01 16:32:25.191175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.190966 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" event={"ID":"4f2fdfa4-7277-41fe-bbf9-97f81116ad25","Type":"ContainerStarted","Data":"b3411aa81030749fb5d06899ca979a87b1dd6659072efc346a25f3f1e2020832"} Nov 01 16:32:25.213544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.213496 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" podStartSLOduration=1.213482734 podStartE2EDuration="1.213482734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:25.210584199 +0000 UTC m=+2682.913750136" watchObservedRunningTime="2026-11-01 16:32:25.213482734 +0000 UTC m=+2682.916648710" Nov 01 16:32:25.569849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.569817 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:25.745598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.745556 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:25.787765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.787714 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq"] Nov 01 16:32:25.791641 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.791609 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-f47890-9dgpq"] Nov 01 16:32:25.847043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.846966 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:25.863484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.863460 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8"] Nov 01 16:32:25.878790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.876781 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:25.903313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.902858 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8"] Nov 01 16:32:25.985292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.985260 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c4zn\" (UniqueName: \"kubernetes.io/projected/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-kube-api-access-4c4zn\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:25.985528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.985506 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:25.985579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:25.985563 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:26.079352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.079322 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:26.087026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.086997 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:26.087143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.087059 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:26.087143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.087101 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c4zn\" (UniqueName: \"kubernetes.io/projected/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-kube-api-access-4c4zn\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:26.087840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.087821 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:26.089286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.089262 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:26.094932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.094911 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c4zn\" (UniqueName: \"kubernetes.io/projected/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-kube-api-access-4c4zn\") pod \"maas-api-revoke-sub-ecbb9205-55fp8\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:26.202717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.202680 4573 generic.go:356] "Generic (PLEG): container finished" podID="4f2fdfa4-7277-41fe-bbf9-97f81116ad25" containerID="a470048b500f723a50a8a48e74d78a8fa97dace43d1f27c4021904076931c02e" exitCode=0 Nov 01 16:32:26.202958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.202783 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" event={"ID":"4f2fdfa4-7277-41fe-bbf9-97f81116ad25","Type":"ContainerDied","Data":"a470048b500f723a50a8a48e74d78a8fa97dace43d1f27c4021904076931c02e"} Nov 01 16:32:26.215106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.215081 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:26.257135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.257097 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:26.461341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.461257 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84667fdb-a43f-4878-8ac7-e0a60bb32d06" path="/var/lib/kubelet/pods/84667fdb-a43f-4878-8ac7-e0a60bb32d06/volumes" Nov 01 16:32:26.508860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.508708 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8"] Nov 01 16:32:26.577446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:26.577412 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:27.211965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.211867 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" event={"ID":"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22","Type":"ContainerStarted","Data":"4e60b8c6241e00eb0c74e20b390df9085d41641c78ac8716fa9f9b58eb89255a"} Nov 01 16:32:27.211965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.211923 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" event={"ID":"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22","Type":"ContainerStarted","Data":"b8f305ee85e1001ea2069c784aef405a73d56e76c66b5bd6d812caf9a4c65775"} Nov 01 16:32:27.229381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.229330 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" podStartSLOduration=2.22931727 podStartE2EDuration="2.22931727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:27.227732209 +0000 UTC m=+2684.930898138" watchObservedRunningTime="2026-11-01 16:32:27.22931727 +0000 UTC m=+2684.932483207" Nov 01 16:32:27.260436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.260404 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:27.409574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.409547 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:27.505339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.505251 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-openshift-ca\") pod \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " Nov 01 16:32:27.505533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.505339 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-kube-api-access-xbvbl\" (UniqueName: \"kubernetes.io/projected/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-kube-api-access-xbvbl\") pod \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " Nov 01 16:32:27.505533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.505475 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-xks-ca\") pod \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\" (UID: \"4f2fdfa4-7277-41fe-bbf9-97f81116ad25\") " Nov 01 16:32:27.505711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.505596 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-openshift-ca" pod "4f2fdfa4-7277-41fe-bbf9-97f81116ad25" (UID: "4f2fdfa4-7277-41fe-bbf9-97f81116ad25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:27.506031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.506010 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:27.507676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.507640 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-xks-ca" pod "4f2fdfa4-7277-41fe-bbf9-97f81116ad25" (UID: "4f2fdfa4-7277-41fe-bbf9-97f81116ad25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:27.507840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.507714 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-kube-api-access-xbvbl" pod "4f2fdfa4-7277-41fe-bbf9-97f81116ad25" (UID: "4f2fdfa4-7277-41fe-bbf9-97f81116ad25"). InnerVolumeSpecName "kube-api-access-xbvbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:27.606943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.606913 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbvbl\" (UniqueName: \"kubernetes.io/projected/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-kube-api-access-xbvbl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:27.606943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.606939 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f2fdfa4-7277-41fe-bbf9-97f81116ad25-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:27.937882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:27.937839 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:28.102772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.100353 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:28.208831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.200404 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:28.235448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.235090 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" Nov 01 16:32:28.235448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.235143 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524" event={"ID":"4f2fdfa4-7277-41fe-bbf9-97f81116ad25","Type":"ContainerDied","Data":"b3411aa81030749fb5d06899ca979a87b1dd6659072efc346a25f3f1e2020832"} Nov 01 16:32:28.235448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.235204 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3411aa81030749fb5d06899ca979a87b1dd6659072efc346a25f3f1e2020832" Nov 01 16:32:28.239293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.239199 4573 generic.go:356] "Generic (PLEG): container finished" podID="3b8ef3b4-e34c-45a9-afdc-55c2d2094f22" containerID="4e60b8c6241e00eb0c74e20b390df9085d41641c78ac8716fa9f9b58eb89255a" exitCode=0 Nov 01 16:32:28.239456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.239383 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" event={"ID":"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22","Type":"ContainerDied","Data":"4e60b8c6241e00eb0c74e20b390df9085d41641c78ac8716fa9f9b58eb89255a"} Nov 01 16:32:28.412041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.412004 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:28.575906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.575728 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:28.757691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.757656 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:28.916699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:28.916672 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:32:29.025582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.025542 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t"] Nov 01 16:32:29.032226 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.032200 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-694a0e6a-mcm9t"] Nov 01 16:32:29.037695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.037661 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-socket\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.037987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.037827 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-credential-socket\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.037987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.037907 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-kube-api-access-lj87c\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-kube-api-access-lj87c\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.038176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038002 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-podinfo\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.038176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038033 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-envoy\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.038176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038117 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-certs\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.038176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038147 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-data\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.038396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038242 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44daf16c-1043-4b4e-bb2d-27aef63adf38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44daf16c-1043-4b4e-bb2d-27aef63adf38-istiod-ca-cert\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.038396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038386 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-token\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-token\") pod \"44daf16c-1043-4b4e-bb2d-27aef63adf38\" (UID: \"44daf16c-1043-4b4e-bb2d-27aef63adf38\") " Nov 01 16:32:29.038610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038587 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-credential-socket" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.038740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038717 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-socket" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.038951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038927 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-certs" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.039033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.038977 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-data" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.039262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.039237 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44daf16c-1043-4b4e-bb2d-27aef63adf38-istiod-ca-cert" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:29.039428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.039409 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.039542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.039529 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.039630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.039618 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.039732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.039716 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.041211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.041186 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-token" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.042100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.042074 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-podinfo" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:29.042192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.042119 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-kube-api-access-lj87c" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "kube-api-access-lj87c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.043578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.043541 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-envoy" pod "44daf16c-1043-4b4e-bb2d-27aef63adf38" (UID: "44daf16c-1043-4b4e-bb2d-27aef63adf38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:29.141647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.141568 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.141647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.141595 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj87c\" (UniqueName: \"kubernetes.io/projected/44daf16c-1043-4b4e-bb2d-27aef63adf38-kube-api-access-lj87c\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.141647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.141606 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.141647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.141615 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44daf16c-1043-4b4e-bb2d-27aef63adf38-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.141647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.141625 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44daf16c-1043-4b4e-bb2d-27aef63adf38-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.249832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.249791 4573 generic.go:356] "Generic (PLEG): container finished" podID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerID="c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b" exitCode=0 Nov 01 16:32:29.250029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.249833 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" Nov 01 16:32:29.250029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.249847 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" event={"ID":"44daf16c-1043-4b4e-bb2d-27aef63adf38","Type":"ContainerDied","Data":"c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b"} Nov 01 16:32:29.250029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.249919 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r" event={"ID":"44daf16c-1043-4b4e-bb2d-27aef63adf38","Type":"ContainerDied","Data":"ffbc5b081804e15d4567135c0b168d8ab8f51da3bb862dd7467eb662b17ef130"} Nov 01 16:32:29.250029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.249952 4573 scope.go:122] "RemoveContainer" containerID="c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b" Nov 01 16:32:29.263713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.263688 4573 scope.go:122] "RemoveContainer" containerID="c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b" Nov 01 16:32:29.266154 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:32:29.266118 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b\": container with ID starting with c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b not found: ID does not exist" containerID="c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b" Nov 01 16:32:29.266281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.266165 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b"} err="failed to get container status \"c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b\": rpc error: code = NotFound desc = could not find container \"c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b\": container with ID starting with c6636300e25cb819cd878feb8b8a08dcea0601f2cc0be34dd6973395abcf465b not found: ID does not exist" Nov 01 16:32:29.286552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.286513 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r"] Nov 01 16:32:29.291839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.291808 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-028c46-openshift-default-5c6465b9f7-bh45r"] Nov 01 16:32:29.293743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.293720 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:29.424444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.424411 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:29.562059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.562025 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-openshift-ca\") pod \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " Nov 01 16:32:29.562210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.562081 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-xks-ca\") pod \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " Nov 01 16:32:29.562210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.562122 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-kube-api-access-4c4zn\" (UniqueName: \"kubernetes.io/projected/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-kube-api-access-4c4zn\") pod \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\" (UID: \"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22\") " Nov 01 16:32:29.562384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.562363 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-openshift-ca" pod "3b8ef3b4-e34c-45a9-afdc-55c2d2094f22" (UID: "3b8ef3b4-e34c-45a9-afdc-55c2d2094f22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:29.562723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.562696 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.564273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.564255 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-xks-ca" pod "3b8ef3b4-e34c-45a9-afdc-55c2d2094f22" (UID: "3b8ef3b4-e34c-45a9-afdc-55c2d2094f22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:29.564273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.564262 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-kube-api-access-4c4zn" pod "3b8ef3b4-e34c-45a9-afdc-55c2d2094f22" (UID: "3b8ef3b4-e34c-45a9-afdc-55c2d2094f22"). InnerVolumeSpecName "kube-api-access-4c4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:29.663619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.663548 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:29.663619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:29.663573 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c4zn\" (UniqueName: \"kubernetes.io/projected/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22-kube-api-access-4c4zn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:30.167094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:30.167040 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:30.342636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:30.342602 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" event={"ID":"3b8ef3b4-e34c-45a9-afdc-55c2d2094f22","Type":"ContainerDied","Data":"b8f305ee85e1001ea2069c784aef405a73d56e76c66b5bd6d812caf9a4c65775"} Nov 01 16:32:30.342929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:30.342911 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f305ee85e1001ea2069c784aef405a73d56e76c66b5bd6d812caf9a4c65775" Nov 01 16:32:30.343090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:30.343066 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8" Nov 01 16:32:30.363829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:30.363206 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" podUID="31dd6eb6-7339-4276-8332-e89ec52d1d5e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:32:30.460942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:30.460861 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" path="/var/lib/kubelet/pods/44daf16c-1043-4b4e-bb2d-27aef63adf38/volumes" Nov 01 16:32:30.461402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:30.461377 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ad9212-eac0-44ea-a2af-0da2b25c2fd5" path="/var/lib/kubelet/pods/76ad9212-eac0-44ea-a2af-0da2b25c2fd5/volumes" Nov 01 16:32:30.994216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:30.994174 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:31.683274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:31.683230 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:32.383110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:32.383075 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:32.484722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:32.484690 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:32.585144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:32.585103 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:32.831861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:32.831779 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll"] Nov 01 16:32:32.832574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:32.832542 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" containerID="cri-o://6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7" gracePeriod=30 Nov 01 16:32:32.868143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:32.868092 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:33.280970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:33.280934 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:34.029056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.028981 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:34.385746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.385708 4573 generic.go:356] "Generic (PLEG): container finished" podID="31dd6eb6-7339-4276-8332-e89ec52d1d5e" containerID="2993fc65690e1d5baa0516df25ea6ea4b50c2ded10775ddeb41e3be76fdbb468" exitCode=0 Nov 01 16:32:34.386068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.386044 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" event={"ID":"31dd6eb6-7339-4276-8332-e89ec52d1d5e","Type":"ContainerDied","Data":"2993fc65690e1d5baa0516df25ea6ea4b50c2ded10775ddeb41e3be76fdbb468"} Nov 01 16:32:34.422527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.422500 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 16:32:34.520821 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.520787 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:34.587373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.585514 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31dd6eb6-7339-4276-8332-e89ec52d1d5e-kube-api-access-jc58v\" (UniqueName: \"kubernetes.io/projected/31dd6eb6-7339-4276-8332-e89ec52d1d5e-kube-api-access-jc58v\") pod \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\" (UID: \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\") " Nov 01 16:32:34.587558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.587381 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31dd6eb6-7339-4276-8332-e89ec52d1d5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd6eb6-7339-4276-8332-e89ec52d1d5e-plugins-config-volume\") pod \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\" (UID: \"31dd6eb6-7339-4276-8332-e89ec52d1d5e\") " Nov 01 16:32:34.588727 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.588668 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31dd6eb6-7339-4276-8332-e89ec52d1d5e-plugins-config-volume" pod "31dd6eb6-7339-4276-8332-e89ec52d1d5e" (UID: "31dd6eb6-7339-4276-8332-e89ec52d1d5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:34.595213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.595161 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31dd6eb6-7339-4276-8332-e89ec52d1d5e-kube-api-access-jc58v" pod "31dd6eb6-7339-4276-8332-e89ec52d1d5e" (UID: "31dd6eb6-7339-4276-8332-e89ec52d1d5e"). InnerVolumeSpecName "kube-api-access-jc58v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:34.630039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.622051 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:34.689779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.689723 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc58v\" (UniqueName: \"kubernetes.io/projected/31dd6eb6-7339-4276-8332-e89ec52d1d5e-kube-api-access-jc58v\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:34.689779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.689784 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31dd6eb6-7339-4276-8332-e89ec52d1d5e-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:34.945268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:34.945227 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:35.054467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.054430 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:35.396258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.396233 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" Nov 01 16:32:35.396416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.396265 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs" event={"ID":"31dd6eb6-7339-4276-8332-e89ec52d1d5e","Type":"ContainerDied","Data":"d74029debd527bf23bf71a093fec333eb1dc8939c3d557242b4a6c3faed8ac93"} Nov 01 16:32:35.396416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.396359 4573 scope.go:122] "RemoveContainer" containerID="2993fc65690e1d5baa0516df25ea6ea4b50c2ded10775ddeb41e3be76fdbb468" Nov 01 16:32:35.425539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.425514 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs"] Nov 01 16:32:35.430344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.430318 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-43fac8-86d989f9c-jnlzs"] Nov 01 16:32:35.457829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.457805 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:35.559469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.559436 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:35.789617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:35.789525 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:36.059687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.059616 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p"] Nov 01 16:32:36.060319 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060303 4573 cpu_manager.go:414] "containerMap: removing container" podUID="4f2fdfa4-7277-41fe-bbf9-97f81116ad25" containerName="revoke-keys" Nov 01 16:32:36.060364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060321 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2fdfa4-7277-41fe-bbf9-97f81116ad25" containerName="revoke-keys" Nov 01 16:32:36.060364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060337 4573 cpu_manager.go:414] "containerMap: removing container" podUID="31dd6eb6-7339-4276-8332-e89ec52d1d5e" containerName="payload-processing" Nov 01 16:32:36.060364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060342 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31dd6eb6-7339-4276-8332-e89ec52d1d5e" containerName="payload-processing" Nov 01 16:32:36.060469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060373 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3b8ef3b4-e34c-45a9-afdc-55c2d2094f22" containerName="revoke-keys" Nov 01 16:32:36.060469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060378 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b8ef3b4-e34c-45a9-afdc-55c2d2094f22" containerName="revoke-keys" Nov 01 16:32:36.060469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060387 4573 cpu_manager.go:414] "containerMap: removing container" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" Nov 01 16:32:36.060469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060391 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" Nov 01 16:32:36.060588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060472 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="44daf16c-1043-4b4e-bb2d-27aef63adf38" containerName="istio-proxy" Nov 01 16:32:36.060588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060487 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2fdfa4-7277-41fe-bbf9-97f81116ad25" containerName="revoke-keys" Nov 01 16:32:36.060588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060494 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="31dd6eb6-7339-4276-8332-e89ec52d1d5e" containerName="payload-processing" Nov 01 16:32:36.060588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.060505 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b8ef3b4-e34c-45a9-afdc-55c2d2094f22" containerName="revoke-keys" Nov 01 16:32:36.065704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.065609 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.069324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.069304 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:32:36.069634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.069601 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:32:36.070353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.070331 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p"] Nov 01 16:32:36.232730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.232699 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt8kr\" (UniqueName: \"kubernetes.io/projected/91f34694-5902-4ef4-bc9d-97ddb9e083f1-kube-api-access-bt8kr\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.232937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.232808 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.232937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.232885 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.334053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.333952 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt8kr\" (UniqueName: \"kubernetes.io/projected/91f34694-5902-4ef4-bc9d-97ddb9e083f1-kube-api-access-bt8kr\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.334053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.334049 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.334299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.334110 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.334935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.334908 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.336820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.336793 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.342224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.342195 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt8kr\" (UniqueName: \"kubernetes.io/projected/91f34694-5902-4ef4-bc9d-97ddb9e083f1-kube-api-access-bt8kr\") pod \"maas-api-revoke-sub-7eca9fd4-zsp8p\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.352167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.352120 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:36.385261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.385235 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:36.512805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.512771 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31dd6eb6-7339-4276-8332-e89ec52d1d5e" path="/var/lib/kubelet/pods/31dd6eb6-7339-4276-8332-e89ec52d1d5e/volumes" Nov 01 16:32:36.680489 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:36.680452 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f34694_5902_4ef4_bc9d_97ddb9e083f1.slice/crio-ba04ead5a8dd8256a1f64d0c9cfcbc460fe9e8439c12a3727e6b646003ef11fc WatchSource:0}: Error finding container ba04ead5a8dd8256a1f64d0c9cfcbc460fe9e8439c12a3727e6b646003ef11fc: Status 404 returned error can't find the container with id ba04ead5a8dd8256a1f64d0c9cfcbc460fe9e8439c12a3727e6b646003ef11fc Nov 01 16:32:36.680776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:36.680732 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p"] Nov 01 16:32:37.181211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:37.181127 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:37.416903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:37.416868 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" event={"ID":"91f34694-5902-4ef4-bc9d-97ddb9e083f1","Type":"ContainerStarted","Data":"c01b89ea08955af7aaaf9cd3e512e53a052106319ff78f014306ff5b7326c356"} Nov 01 16:32:37.417065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:37.416921 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" event={"ID":"91f34694-5902-4ef4-bc9d-97ddb9e083f1","Type":"ContainerStarted","Data":"ba04ead5a8dd8256a1f64d0c9cfcbc460fe9e8439c12a3727e6b646003ef11fc"} Nov 01 16:32:37.433171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:37.433084 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" podStartSLOduration=1.43307235 podStartE2EDuration="1.43307235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:37.430577594 +0000 UTC m=+2695.133743530" watchObservedRunningTime="2026-11-01 16:32:37.43307235 +0000 UTC m=+2695.136238286" Nov 01 16:32:38.118540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.118517 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:32:38.212474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.212437 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.266907 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-kube-api-access-s68sq\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-kube-api-access-s68sq\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.266984 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86900094-5677-4a05-b22e-f06a98775de6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86900094-5677-4a05-b22e-f06a98775de6-istio-podinfo\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.267060 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-envoy\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.267137 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-istio-token\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-istio-token\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.267420 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86900094-5677-4a05-b22e-f06a98775de6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86900094-5677-4a05-b22e-f06a98775de6-istiod-ca-cert\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.268136 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-socket\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.268216 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-data\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.268246 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-certs\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.268364 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-credential-socket\") pod \"86900094-5677-4a05-b22e-f06a98775de6\" (UID: \"86900094-5677-4a05-b22e-f06a98775de6\") " Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.269774 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-credential-socket" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.271254 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-socket" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.273427 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-data" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.273628 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-certs" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.277255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.274086 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86900094-5677-4a05-b22e-f06a98775de6-istiod-ca-cert" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:38.291070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.290973 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86900094-5677-4a05-b22e-f06a98775de6-istio-podinfo" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:32:38.291723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.291683 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-envoy" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:32:38.291723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.291684 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-kube-api-access-s68sq" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "kube-api-access-s68sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:38.294059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.293998 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-istio-token" pod "86900094-5677-4a05-b22e-f06a98775de6" (UID: "86900094-5677-4a05-b22e-f06a98775de6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:38.370898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370822 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86900094-5677-4a05-b22e-f06a98775de6-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.370898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370852 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.370898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370861 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.370898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370872 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.370898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370881 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.370898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370890 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s68sq\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-kube-api-access-s68sq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.370898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370900 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86900094-5677-4a05-b22e-f06a98775de6-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.371264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370909 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86900094-5677-4a05-b22e-f06a98775de6-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.371264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.370918 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86900094-5677-4a05-b22e-f06a98775de6-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:38.429037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.429002 4573 generic.go:356] "Generic (PLEG): container finished" podID="91f34694-5902-4ef4-bc9d-97ddb9e083f1" containerID="c01b89ea08955af7aaaf9cd3e512e53a052106319ff78f014306ff5b7326c356" exitCode=0 Nov 01 16:32:38.429201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.429090 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" event={"ID":"91f34694-5902-4ef4-bc9d-97ddb9e083f1","Type":"ContainerDied","Data":"c01b89ea08955af7aaaf9cd3e512e53a052106319ff78f014306ff5b7326c356"} Nov 01 16:32:38.430726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.430702 4573 generic.go:356] "Generic (PLEG): container finished" podID="86900094-5677-4a05-b22e-f06a98775de6" containerID="6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7" exitCode=0 Nov 01 16:32:38.430837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.430760 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" event={"ID":"86900094-5677-4a05-b22e-f06a98775de6","Type":"ContainerDied","Data":"6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7"} Nov 01 16:32:38.430837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.430767 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" Nov 01 16:32:38.430837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.430801 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll" event={"ID":"86900094-5677-4a05-b22e-f06a98775de6","Type":"ContainerDied","Data":"991a9d55bdb06ce0dcea1562ecf518b5f8a64f8848930f0a12c14044d60ac3ee"} Nov 01 16:32:38.430837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.430828 4573 scope.go:122] "RemoveContainer" containerID="6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7" Nov 01 16:32:38.444400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.444369 4573 scope.go:122] "RemoveContainer" containerID="6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7" Nov 01 16:32:38.444637 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:32:38.444621 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7\": container with ID starting with 6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7 not found: ID does not exist" containerID="6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7" Nov 01 16:32:38.444676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.444646 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7"} err="failed to get container status \"6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7\": rpc error: code = NotFound desc = could not find container \"6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7\": container with ID starting with 6a212490b6a1e139675d7e873896658f7b2c1bf7be6c7632c50685574bb7b9d7 not found: ID does not exist" Nov 01 16:32:38.476364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.476313 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll"] Nov 01 16:32:38.482326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.482296 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-f47890-openshift-default-557cbcdc57-cjpll"] Nov 01 16:32:38.491636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.491583 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:38.594089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:38.594029 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:39.179978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.179946 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:39.607030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.606997 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:39.689116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.689081 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:39.790035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.789938 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-xks-ca\") pod \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " Nov 01 16:32:39.790035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.790009 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91f34694-5902-4ef4-bc9d-97ddb9e083f1-kube-api-access-bt8kr\" (UniqueName: \"kubernetes.io/projected/91f34694-5902-4ef4-bc9d-97ddb9e083f1-kube-api-access-bt8kr\") pod \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " Nov 01 16:32:39.790291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.790076 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-openshift-ca\") pod \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\" (UID: \"91f34694-5902-4ef4-bc9d-97ddb9e083f1\") " Nov 01 16:32:39.790475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.790451 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-openshift-ca" pod "91f34694-5902-4ef4-bc9d-97ddb9e083f1" (UID: "91f34694-5902-4ef4-bc9d-97ddb9e083f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:39.790614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.790596 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:39.792102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.792076 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f34694-5902-4ef4-bc9d-97ddb9e083f1-kube-api-access-bt8kr" pod "91f34694-5902-4ef4-bc9d-97ddb9e083f1" (UID: "91f34694-5902-4ef4-bc9d-97ddb9e083f1"). InnerVolumeSpecName "kube-api-access-bt8kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:39.792102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.792088 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-xks-ca" pod "91f34694-5902-4ef4-bc9d-97ddb9e083f1" (UID: "91f34694-5902-4ef4-bc9d-97ddb9e083f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:39.794024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.793977 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:39.898786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.897775 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91f34694-5902-4ef4-bc9d-97ddb9e083f1-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:39.898786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:39.897819 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt8kr\" (UniqueName: \"kubernetes.io/projected/91f34694-5902-4ef4-bc9d-97ddb9e083f1-kube-api-access-bt8kr\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:40.370252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:40.370221 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:40.449465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:40.449438 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" Nov 01 16:32:40.449645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:40.449436 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p" event={"ID":"91f34694-5902-4ef4-bc9d-97ddb9e083f1","Type":"ContainerDied","Data":"ba04ead5a8dd8256a1f64d0c9cfcbc460fe9e8439c12a3727e6b646003ef11fc"} Nov 01 16:32:40.449645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:40.449561 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba04ead5a8dd8256a1f64d0c9cfcbc460fe9e8439c12a3727e6b646003ef11fc" Nov 01 16:32:40.459858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:40.459828 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86900094-5677-4a05-b22e-f06a98775de6" path="/var/lib/kubelet/pods/86900094-5677-4a05-b22e-f06a98775de6/volumes" Nov 01 16:32:40.909745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:40.909699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:41.018785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:41.018377 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:41.327617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:41.327541 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:41.808929 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:41.808897 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:42.078806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:42.078683 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:43.356847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:43.356815 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:32:43.368143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:43.368118 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:32:43.369222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:43.369203 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:32:43.384039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:43.384020 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:32:43.432551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:43.432519 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:43.528379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:43.528335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:43.648828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:43.642367 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:44.397696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:44.397657 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:44.941726 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:44.941695 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6"] Nov 01 16:32:44.941950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:44.941906 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" podUID="dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" containerName="maas-api" containerID="cri-o://fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67" gracePeriod=30 Nov 01 16:32:44.972420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:44.972391 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:45.058325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.058260 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:45.131631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.131565 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:45.221932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.221899 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524"] Nov 01 16:32:45.223005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.222973 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:45.226022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.225999 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-aa6303-pf524"] Nov 01 16:32:45.237196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.237171 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:32:45.276909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.276881 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-kube-api-access-pmsmd\" (UniqueName: \"kubernetes.io/projected/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-kube-api-access-pmsmd\") pod \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " Nov 01 16:32:45.277075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.276966 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-metrics-certs\") pod \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " Nov 01 16:32:45.277075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.276997 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-openshift-service-ca\") pod \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " Nov 01 16:32:45.277198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.277117 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-maas-api-tls\") pod \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\" (UID: \"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b\") " Nov 01 16:32:45.277571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.277545 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-openshift-service-ca" pod "dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" (UID: "dc61b142-ba1c-4fa4-b93f-95b880e2ae9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:45.279531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.279506 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-maas-api-tls" pod "dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" (UID: "dc61b142-ba1c-4fa4-b93f-95b880e2ae9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:45.279656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.279633 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-metrics-certs" pod "dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" (UID: "dc61b142-ba1c-4fa4-b93f-95b880e2ae9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:45.279788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.279733 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-kube-api-access-pmsmd" pod "dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" (UID: "dc61b142-ba1c-4fa4-b93f-95b880e2ae9b"). InnerVolumeSpecName "kube-api-access-pmsmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:45.307721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.307681 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:45.397775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.396622 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmsmd\" (UniqueName: \"kubernetes.io/projected/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-kube-api-access-pmsmd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:45.397775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.396662 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:45.397775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.396679 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:45.397775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.396693 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:45.424793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.414193 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:45.501495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.501417 4573 generic.go:356] "Generic (PLEG): container finished" podID="dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" containerID="fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67" exitCode=0 Nov 01 16:32:45.501495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.501469 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" event={"ID":"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b","Type":"ContainerDied","Data":"fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67"} Nov 01 16:32:45.501495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.501477 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" Nov 01 16:32:45.501823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.501510 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6" event={"ID":"dc61b142-ba1c-4fa4-b93f-95b880e2ae9b","Type":"ContainerDied","Data":"3358e6ebbd99c748c3b266009c13880cac67009ebe66ed13905b3e2f1add0e28"} Nov 01 16:32:45.501823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.501535 4573 scope.go:122] "RemoveContainer" containerID="fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67" Nov 01 16:32:45.524160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.524132 4573 scope.go:122] "RemoveContainer" containerID="fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67" Nov 01 16:32:45.524511 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:32:45.524489 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67\": container with ID starting with fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67 not found: ID does not exist" containerID="fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67" Nov 01 16:32:45.524594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.524520 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67"} err="failed to get container status \"fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67\": rpc error: code = NotFound desc = could not find container \"fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67\": container with ID starting with fa020867d372f5caaa8a830a440dc57405c43768ddb451a813db594ed5b3ee67 not found: ID does not exist" Nov 01 16:32:45.544720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.544687 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6"] Nov 01 16:32:45.548192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.548171 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-aa6303-7c7586566c-5bjl6"] Nov 01 16:32:45.680461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:45.680416 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:46.445382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:46.445349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:46.459195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:46.459167 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2fdfa4-7277-41fe-bbf9-97f81116ad25" path="/var/lib/kubelet/pods/4f2fdfa4-7277-41fe-bbf9-97f81116ad25/volumes" Nov 01 16:32:46.459736 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:46.459714 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" path="/var/lib/kubelet/pods/dc61b142-ba1c-4fa4-b93f-95b880e2ae9b/volumes" Nov 01 16:32:47.200627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:47.200591 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:48.079944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:48.079889 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:49.294391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:49.294351 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:49.348255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:49.348226 4573 scope.go:122] "RemoveContainer" containerID="1399703c91306f69adf7eef740b496adcece8e4b6a26d1586944e83a431f9d7d" Nov 01 16:32:49.358598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:49.358575 4573 scope.go:122] "RemoveContainer" containerID="15c1ac11d0da1767b5775b31341cedd985c2d1505f4ff570131ee74cf0ae6d2b" Nov 01 16:32:49.407889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:49.407853 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:49.495374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:49.495347 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:49.596029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:49.595950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:49.694962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:49.694927 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:50.250994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:50.250956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:51.252926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:51.252896 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:51.752885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:51.752849 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:51.871493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:51.871464 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:52.110462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.110427 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:52.207547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.207517 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:52.349101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.349067 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:52.540098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.540019 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:52.677764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.677727 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7"] Nov 01 16:32:52.678396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678378 4573 cpu_manager.go:414] "containerMap: removing container" podUID="dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" containerName="maas-api" Nov 01 16:32:52.678396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678396 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" containerName="maas-api" Nov 01 16:32:52.678537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678424 4573 cpu_manager.go:414] "containerMap: removing container" podUID="91f34694-5902-4ef4-bc9d-97ddb9e083f1" containerName="revoke-keys" Nov 01 16:32:52.678537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678429 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91f34694-5902-4ef4-bc9d-97ddb9e083f1" containerName="revoke-keys" Nov 01 16:32:52.678537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678460 4573 cpu_manager.go:414] "containerMap: removing container" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" Nov 01 16:32:52.678537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678466 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" Nov 01 16:32:52.678671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678544 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="91f34694-5902-4ef4-bc9d-97ddb9e083f1" containerName="revoke-keys" Nov 01 16:32:52.678671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678558 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="86900094-5677-4a05-b22e-f06a98775de6" containerName="istio-proxy" Nov 01 16:32:52.678671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.678565 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc61b142-ba1c-4fa4-b93f-95b880e2ae9b" containerName="maas-api" Nov 01 16:32:52.684333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.684305 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.688602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.688581 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7"] Nov 01 16:32:52.723088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.723058 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:32:52.723254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.723236 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:32:52.761618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.761585 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:52.818934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.818853 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrqd\" (UniqueName: \"kubernetes.io/projected/2d10ba85-9fe1-4f4b-970b-825f667c1649-kube-api-access-kdrqd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.819101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.819053 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.819175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.819101 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.920740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.920702 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.920740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.920802 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.921087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.920977 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrqd\" (UniqueName: \"kubernetes.io/projected/2d10ba85-9fe1-4f4b-970b-825f667c1649-kube-api-access-kdrqd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.921473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.921446 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.923313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.923289 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:52.931050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:52.931026 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrqd\" (UniqueName: \"kubernetes.io/projected/2d10ba85-9fe1-4f4b-970b-825f667c1649-kube-api-access-kdrqd\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:53.002636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:53.002608 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:53.036134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:53.036104 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:53.290553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:53.290524 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7"] Nov 01 16:32:53.291150 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:53.291123 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d10ba85_9fe1_4f4b_970b_825f667c1649.slice/crio-a9dfb9fd89d9faa64fdf255ecec4930be566462f21e8fa333915d07a592c5318 WatchSource:0}: Error finding container a9dfb9fd89d9faa64fdf255ecec4930be566462f21e8fa333915d07a592c5318: Status 404 returned error can't find the container with id a9dfb9fd89d9faa64fdf255ecec4930be566462f21e8fa333915d07a592c5318 Nov 01 16:32:53.586322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:53.586287 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" event={"ID":"2d10ba85-9fe1-4f4b-970b-825f667c1649","Type":"ContainerStarted","Data":"197a004a771ee8894cea1a15bf22f3c6514d8d1739cf98179966e977d69dd805"} Nov 01 16:32:53.586637 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:53.586343 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" event={"ID":"2d10ba85-9fe1-4f4b-970b-825f667c1649","Type":"ContainerStarted","Data":"a9dfb9fd89d9faa64fdf255ecec4930be566462f21e8fa333915d07a592c5318"} Nov 01 16:32:53.602392 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:53.602347 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" podStartSLOduration=1.602334159 podStartE2EDuration="1.602334159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:32:53.599885869 +0000 UTC m=+2711.303051806" watchObservedRunningTime="2026-11-01 16:32:53.602334159 +0000 UTC m=+2711.305500091" Nov 01 16:32:53.754728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:53.754679 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:54.364699 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:32:54.364666 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:32:54.596311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.596278 4573 generic.go:356] "Generic (PLEG): container finished" podID="2d10ba85-9fe1-4f4b-970b-825f667c1649" containerID="197a004a771ee8894cea1a15bf22f3c6514d8d1739cf98179966e977d69dd805" exitCode=0 Nov 01 16:32:54.596680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.596345 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" event={"ID":"2d10ba85-9fe1-4f4b-970b-825f667c1649","Type":"ContainerDied","Data":"197a004a771ee8894cea1a15bf22f3c6514d8d1739cf98179966e977d69dd805"} Nov 01 16:32:54.598376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.598345 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:54.700829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.687183 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:54.742876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.742843 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q"] Nov 01 16:32:54.747229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.747191 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.755144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.755118 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q"] Nov 01 16:32:54.868534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.868505 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrwt\" (UniqueName: \"kubernetes.io/projected/8af9bdb9-8281-425a-adab-1f716019b7cb-kube-api-access-wtrwt\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.868799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.868539 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.868883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.868800 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.970398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.970324 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrwt\" (UniqueName: \"kubernetes.io/projected/8af9bdb9-8281-425a-adab-1f716019b7cb-kube-api-access-wtrwt\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.970398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.970361 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.970643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.970505 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.971155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.971123 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.973006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.972983 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:54.977773 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:54.977734 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrwt\" (UniqueName: \"kubernetes.io/projected/8af9bdb9-8281-425a-adab-1f716019b7cb-kube-api-access-wtrwt\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:55.071569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.071539 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:55.349220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.349190 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q"] Nov 01 16:32:55.349992 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:32:55.349954 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8af9bdb9_8281_425a_adab_1f716019b7cb.slice/crio-04604c433ec26185a59186d37091cead299801198d4c1a0c3a96f898c5f8a6fa WatchSource:0}: Error finding container 04604c433ec26185a59186d37091cead299801198d4c1a0c3a96f898c5f8a6fa: Status 404 returned error can't find the container with id 04604c433ec26185a59186d37091cead299801198d4c1a0c3a96f898c5f8a6fa Nov 01 16:32:55.605038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.604967 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" event={"ID":"8af9bdb9-8281-425a-adab-1f716019b7cb","Type":"ContainerStarted","Data":"04604c433ec26185a59186d37091cead299801198d4c1a0c3a96f898c5f8a6fa"} Nov 01 16:32:55.634450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.634018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:55.773365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.773346 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:55.883199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.883163 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d10ba85-9fe1-4f4b-970b-825f667c1649-kube-api-access-kdrqd\" (UniqueName: \"kubernetes.io/projected/2d10ba85-9fe1-4f4b-970b-825f667c1649-kube-api-access-kdrqd\") pod \"2d10ba85-9fe1-4f4b-970b-825f667c1649\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " Nov 01 16:32:55.883374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.883281 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-xks-ca\") pod \"2d10ba85-9fe1-4f4b-970b-825f667c1649\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " Nov 01 16:32:55.883374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.883330 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-openshift-ca\") pod \"2d10ba85-9fe1-4f4b-970b-825f667c1649\" (UID: \"2d10ba85-9fe1-4f4b-970b-825f667c1649\") " Nov 01 16:32:55.883731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.883696 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-openshift-ca" pod "2d10ba85-9fe1-4f4b-970b-825f667c1649" (UID: "2d10ba85-9fe1-4f4b-970b-825f667c1649"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:55.883902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.883844 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:55.885407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.885378 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-xks-ca" pod "2d10ba85-9fe1-4f4b-970b-825f667c1649" (UID: "2d10ba85-9fe1-4f4b-970b-825f667c1649"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:55.885526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.885409 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d10ba85-9fe1-4f4b-970b-825f667c1649-kube-api-access-kdrqd" pod "2d10ba85-9fe1-4f4b-970b-825f667c1649" (UID: "2d10ba85-9fe1-4f4b-970b-825f667c1649"). InnerVolumeSpecName "kube-api-access-kdrqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:55.985161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.985128 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d10ba85-9fe1-4f4b-970b-825f667c1649-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:55.985282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:55.985165 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdrqd\" (UniqueName: \"kubernetes.io/projected/2d10ba85-9fe1-4f4b-970b-825f667c1649-kube-api-access-kdrqd\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:56.024656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.024621 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t"] Nov 01 16:32:56.028561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.028526 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a9ad2c91-bw75t"] Nov 01 16:32:56.459363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.459328 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65901982-a0f2-4b25-9efb-22f85a8eb245" path="/var/lib/kubelet/pods/65901982-a0f2-4b25-9efb-22f85a8eb245/volumes" Nov 01 16:32:56.576581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.576490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:56.613359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.613328 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" Nov 01 16:32:56.613893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.613364 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7" event={"ID":"2d10ba85-9fe1-4f4b-970b-825f667c1649","Type":"ContainerDied","Data":"a9dfb9fd89d9faa64fdf255ecec4930be566462f21e8fa333915d07a592c5318"} Nov 01 16:32:56.613893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.613410 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9dfb9fd89d9faa64fdf255ecec4930be566462f21e8fa333915d07a592c5318" Nov 01 16:32:56.615306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.615275 4573 generic.go:356] "Generic (PLEG): container finished" podID="8af9bdb9-8281-425a-adab-1f716019b7cb" containerID="1df6001a76d3e3aa6bce9cda586df0f9ef9f54044d6fb6f388df8109f032b580" exitCode=0 Nov 01 16:32:56.615533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:56.615400 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" event={"ID":"8af9bdb9-8281-425a-adab-1f716019b7cb","Type":"ContainerDied","Data":"1df6001a76d3e3aa6bce9cda586df0f9ef9f54044d6fb6f388df8109f032b580"} Nov 01 16:32:57.250205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:57.250175 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:57.888514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:57.888457 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:58.019322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.019288 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-openshift-ca\") pod \"8af9bdb9-8281-425a-adab-1f716019b7cb\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " Nov 01 16:32:58.019528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.019393 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8af9bdb9-8281-425a-adab-1f716019b7cb-kube-api-access-wtrwt\" (UniqueName: \"kubernetes.io/projected/8af9bdb9-8281-425a-adab-1f716019b7cb-kube-api-access-wtrwt\") pod \"8af9bdb9-8281-425a-adab-1f716019b7cb\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " Nov 01 16:32:58.019528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.019474 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-xks-ca\") pod \"8af9bdb9-8281-425a-adab-1f716019b7cb\" (UID: \"8af9bdb9-8281-425a-adab-1f716019b7cb\") " Nov 01 16:32:58.019723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.019697 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-openshift-ca" pod "8af9bdb9-8281-425a-adab-1f716019b7cb" (UID: "8af9bdb9-8281-425a-adab-1f716019b7cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:32:58.019934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.019914 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:58.021477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.021454 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-xks-ca" pod "8af9bdb9-8281-425a-adab-1f716019b7cb" (UID: "8af9bdb9-8281-425a-adab-1f716019b7cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:32:58.021591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.021516 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8af9bdb9-8281-425a-adab-1f716019b7cb-kube-api-access-wtrwt" pod "8af9bdb9-8281-425a-adab-1f716019b7cb" (UID: "8af9bdb9-8281-425a-adab-1f716019b7cb"). InnerVolumeSpecName "kube-api-access-wtrwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:32:58.121480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.121443 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrwt\" (UniqueName: \"kubernetes.io/projected/8af9bdb9-8281-425a-adab-1f716019b7cb-kube-api-access-wtrwt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:58.121480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.121483 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8af9bdb9-8281-425a-adab-1f716019b7cb-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:32:58.128150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.128118 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:58.632040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.632008 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" Nov 01 16:32:58.632040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.632023 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q" event={"ID":"8af9bdb9-8281-425a-adab-1f716019b7cb","Type":"ContainerDied","Data":"04604c433ec26185a59186d37091cead299801198d4c1a0c3a96f898c5f8a6fa"} Nov 01 16:32:58.632270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.632073 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04604c433ec26185a59186d37091cead299801198d4c1a0c3a96f898c5f8a6fa" Nov 01 16:32:58.980099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:58.980070 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:59.387097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:59.387065 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:59.488683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:59.488652 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:59.782504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:59.782409 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df"] Nov 01 16:32:59.782682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:59.782652 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" containerID="cri-o://b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c" gracePeriod=30 Nov 01 16:32:59.825384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:59.825353 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:32:59.911470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:32:59.911441 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:00.156698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.156665 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:00.319830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.319800 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf"] Nov 01 16:33:00.320470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.320454 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8af9bdb9-8281-425a-adab-1f716019b7cb" containerName="revoke-keys" Nov 01 16:33:00.320510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.320473 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8af9bdb9-8281-425a-adab-1f716019b7cb" containerName="revoke-keys" Nov 01 16:33:00.320510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.320497 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2d10ba85-9fe1-4f4b-970b-825f667c1649" containerName="revoke-keys" Nov 01 16:33:00.320510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.320503 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d10ba85-9fe1-4f4b-970b-825f667c1649" containerName="revoke-keys" Nov 01 16:33:00.320626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.320617 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8af9bdb9-8281-425a-adab-1f716019b7cb" containerName="revoke-keys" Nov 01 16:33:00.320663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.320631 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d10ba85-9fe1-4f4b-970b-825f667c1649" containerName="revoke-keys" Nov 01 16:33:00.324685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.324659 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.333038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.333013 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf"] Nov 01 16:33:00.409245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.409174 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:33:00.409427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.409404 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:33:00.469891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.469858 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9sh\" (UniqueName: \"kubernetes.io/projected/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-kube-api-access-7t9sh\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.470079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.469914 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.470079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.470020 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.571070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.571042 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9sh\" (UniqueName: \"kubernetes.io/projected/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-kube-api-access-7t9sh\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.571234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.571123 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.571340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.571281 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.571787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.571746 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.573561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.573541 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.579068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.579046 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9sh\" (UniqueName: \"kubernetes.io/projected/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-kube-api-access-7t9sh\") pod \"maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.646318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.646291 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:00.758162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.758129 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:00.929794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:00.926144 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf"] Nov 01 16:33:01.347006 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:01.346968 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:33:01.347429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:01.347039 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:33:01.656349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:01.656309 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" event={"ID":"a431f4fc-4c91-49b0-8ee4-a7bff87f8580","Type":"ContainerStarted","Data":"2eaf3ad402d96ebb539bba781306a154c5b31a0168b251d88a8fdc032e0742cd"} Nov 01 16:33:01.656554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:01.656373 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" event={"ID":"a431f4fc-4c91-49b0-8ee4-a7bff87f8580","Type":"ContainerStarted","Data":"84fdaf3d9effb2e94a9eb5b8f66e13e00198a4f040cae47412a9559067dfbdc6"} Nov 01 16:33:01.678818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:01.678744 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" podStartSLOduration=1.678730305 podStartE2EDuration="1.678730305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:01.675581905 +0000 UTC m=+2719.378747854" watchObservedRunningTime="2026-11-01 16:33:01.678730305 +0000 UTC m=+2719.381896240" Nov 01 16:33:01.771520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:01.771484 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:02.633766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:02.633717 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:02.664622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:02.664596 4573 generic.go:356] "Generic (PLEG): container finished" podID="a431f4fc-4c91-49b0-8ee4-a7bff87f8580" containerID="2eaf3ad402d96ebb539bba781306a154c5b31a0168b251d88a8fdc032e0742cd" exitCode=0 Nov 01 16:33:02.664812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:02.664685 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" event={"ID":"a431f4fc-4c91-49b0-8ee4-a7bff87f8580","Type":"ContainerDied","Data":"2eaf3ad402d96ebb539bba781306a154c5b31a0168b251d88a8fdc032e0742cd"} Nov 01 16:33:02.845683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:02.845649 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4"] Nov 01 16:33:02.859123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:02.859091 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs"] Nov 01 16:33:02.926172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:02.926090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:03.016382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:03.016340 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:03.422604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:03.422572 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:03.656267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:03.656222 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:03.738847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:03.738735 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:03.898065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:03.897732 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:03.909457 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:03.909412 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:04.039415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.039329 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-kube-api-access-7t9sh\" (UniqueName: \"kubernetes.io/projected/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-kube-api-access-7t9sh\") pod \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " Nov 01 16:33:04.039606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.039492 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-xks-ca\") pod \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " Nov 01 16:33:04.039606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.039539 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-openshift-ca\") pod \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\" (UID: \"a431f4fc-4c91-49b0-8ee4-a7bff87f8580\") " Nov 01 16:33:04.040014 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.039986 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-openshift-ca" pod "a431f4fc-4c91-49b0-8ee4-a7bff87f8580" (UID: "a431f4fc-4c91-49b0-8ee4-a7bff87f8580"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:04.040197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.040086 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:04.041561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.041532 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-xks-ca" pod "a431f4fc-4c91-49b0-8ee4-a7bff87f8580" (UID: "a431f4fc-4c91-49b0-8ee4-a7bff87f8580"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:04.041670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.041568 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-kube-api-access-7t9sh" pod "a431f4fc-4c91-49b0-8ee4-a7bff87f8580" (UID: "a431f4fc-4c91-49b0-8ee4-a7bff87f8580"). InnerVolumeSpecName "kube-api-access-7t9sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:04.141849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.141813 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:04.141849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.141848 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t9sh\" (UniqueName: \"kubernetes.io/projected/a431f4fc-4c91-49b0-8ee4-a7bff87f8580-kube-api-access-7t9sh\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:04.223164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.223127 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:04.682033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.682000 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" Nov 01 16:33:04.682033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.682025 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf" event={"ID":"a431f4fc-4c91-49b0-8ee4-a7bff87f8580","Type":"ContainerDied","Data":"84fdaf3d9effb2e94a9eb5b8f66e13e00198a4f040cae47412a9559067dfbdc6"} Nov 01 16:33:04.682606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.682074 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84fdaf3d9effb2e94a9eb5b8f66e13e00198a4f040cae47412a9559067dfbdc6" Nov 01 16:33:04.900860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.899977 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:04.986395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:04.986366 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:05.094807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.094783 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:33:05.202309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.201101 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9"] Nov 01 16:33:05.217341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.217313 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9"] Nov 01 16:33:05.261518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.261447 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:05.271742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.271713 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a241e7-c6e7-4a85-80d4-906fbde4253d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a241e7-c6e7-4a85-80d4-906fbde4253d-istiod-ca-cert\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.272214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.272175 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-certs\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.272462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.272427 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-podinfo\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.272646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.272285 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a241e7-c6e7-4a85-80d4-906fbde4253d-istiod-ca-cert" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:05.272774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.272619 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-certs" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.272952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.272923 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-socket\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.273266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.273248 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-data\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.273525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.273331 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-socket" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.273639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.273499 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-data" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.273795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.273777 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-token\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-token\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.274022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.273993 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-credential-socket\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.274206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.274189 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-envoy\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.274352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.274333 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-kube-api-access-z24s6\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-kube-api-access-z24s6\") pod \"84a241e7-c6e7-4a85-80d4-906fbde4253d\" (UID: \"84a241e7-c6e7-4a85-80d4-906fbde4253d\") " Nov 01 16:33:05.274565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.274544 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-credential-socket" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.275274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.275257 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.275439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.275409 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.275569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.275545 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.275687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.275675 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.275814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.275802 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84a241e7-c6e7-4a85-80d4-906fbde4253d-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.276917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.276895 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-token" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:05.278092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.278071 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-kube-api-access-z24s6" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "kube-api-access-z24s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:05.284949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.284921 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-envoy" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:05.285076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.284996 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-podinfo" pod "84a241e7-c6e7-4a85-80d4-906fbde4253d" (UID: "84a241e7-c6e7-4a85-80d4-906fbde4253d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:05.377157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.377118 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.377157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.377158 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24s6\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-kube-api-access-z24s6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.377380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.377170 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.377380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.377179 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84a241e7-c6e7-4a85-80d4-906fbde4253d-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:05.690791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.690673 4573 generic.go:356] "Generic (PLEG): container finished" podID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerID="b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c" exitCode=0 Nov 01 16:33:05.690791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.690722 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" Nov 01 16:33:05.691259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.690785 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" event={"ID":"84a241e7-c6e7-4a85-80d4-906fbde4253d","Type":"ContainerDied","Data":"b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c"} Nov 01 16:33:05.691259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.690840 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df" event={"ID":"84a241e7-c6e7-4a85-80d4-906fbde4253d","Type":"ContainerDied","Data":"789fa25be4c11a462e62e72aa2892626632335fae377aeca24deb028c689c5cc"} Nov 01 16:33:05.691259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.690870 4573 scope.go:122] "RemoveContainer" containerID="b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c" Nov 01 16:33:05.702854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.702835 4573 scope.go:122] "RemoveContainer" containerID="f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe" Nov 01 16:33:05.715348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.715325 4573 scope.go:122] "RemoveContainer" containerID="b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c" Nov 01 16:33:05.715476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.715455 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df"] Nov 01 16:33:05.715700 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:05.715682 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c\": container with ID starting with b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c not found: ID does not exist" containerID="b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c" Nov 01 16:33:05.715778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.715709 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c"} err="failed to get container status \"b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c\": rpc error: code = NotFound desc = could not find container \"b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c\": container with ID starting with b90b773550f1df4bef8ec2c5bd3a1e3042bd80e93a2477a2675dfe6480e4197c not found: ID does not exist" Nov 01 16:33:05.715778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.715727 4573 scope.go:122] "RemoveContainer" containerID="f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe" Nov 01 16:33:05.716073 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:05.716050 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe\": container with ID starting with f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe not found: ID does not exist" containerID="f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe" Nov 01 16:33:05.716135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.716082 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe"} err="failed to get container status \"f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe\": rpc error: code = NotFound desc = could not find container \"f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe\": container with ID starting with f6b26b3a78de2a44178f0537d733bcd3f0400b517efef9b4acace5fbddbc91fe not found: ID does not exist" Nov 01 16:33:05.735125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.735103 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-734859-openshift-default-d8d8ddd9f-nr6df"] Nov 01 16:33:05.766668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:05.766642 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:06.458917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:06.458882 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" path="/var/lib/kubelet/pods/84a241e7-c6e7-4a85-80d4-906fbde4253d/volumes" Nov 01 16:33:06.471585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:06.471560 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:07.188219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:07.188189 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:07.845941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:07.845898 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" podUID="f82a8513-36f7-4799-a7c7-0c685a52631d" containerName="payload-processing" containerID="cri-o://465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67" gracePeriod=25 Nov 01 16:33:07.859556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:07.859526 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" podUID="fde1a22e-1a29-40fe-ae79-9b3f717fc392" containerName="payload-pre-processing" containerID="cri-o://5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e" gracePeriod=25 Nov 01 16:33:07.872917 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:07.872894 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:08.191926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:08.191882 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" podUID="fde1a22e-1a29-40fe-ae79-9b3f717fc392" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:08.352062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:08.352028 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:33:08.658979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:08.658944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:09.570699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:09.570662 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:10.286659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:10.286627 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:10.458528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:10.458472 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9"] Nov 01 16:33:10.521592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:10.521557 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:10.618038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:10.617999 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:10.780581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:10.780543 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:10.876936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:10.876852 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:11.173018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.172929 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq"] Nov 01 16:33:11.173207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.173181 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" podUID="ec6665a5-bce9-41b5-98ca-d7a46e344e62" containerName="maas-api" containerID="cri-o://1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d" gracePeriod=30 Nov 01 16:33:11.213892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.213858 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:11.342872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.342837 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:11.511195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.511166 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7"] Nov 01 16:33:11.519105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.519084 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-5ccd7b-qbgz7"] Nov 01 16:33:11.535488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.535470 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:33:11.612497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.612470 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec6665a5-bce9-41b5-98ca-d7a46e344e62-kube-api-access-wn4kc\" (UniqueName: \"kubernetes.io/projected/ec6665a5-bce9-41b5-98ca-d7a46e344e62-kube-api-access-wn4kc\") pod \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " Nov 01 16:33:11.612661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.612516 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-metrics-certs\") pod \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " Nov 01 16:33:11.612661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.612557 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-maas-api-tls\") pod \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " Nov 01 16:33:11.612661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.612613 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec6665a5-bce9-41b5-98ca-d7a46e344e62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6665a5-bce9-41b5-98ca-d7a46e344e62-openshift-service-ca\") pod \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\" (UID: \"ec6665a5-bce9-41b5-98ca-d7a46e344e62\") " Nov 01 16:33:11.613132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.613102 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec6665a5-bce9-41b5-98ca-d7a46e344e62-openshift-service-ca" pod "ec6665a5-bce9-41b5-98ca-d7a46e344e62" (UID: "ec6665a5-bce9-41b5-98ca-d7a46e344e62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:11.613310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.613286 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec6665a5-bce9-41b5-98ca-d7a46e344e62-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:11.615082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.615009 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-maas-api-tls" pod "ec6665a5-bce9-41b5-98ca-d7a46e344e62" (UID: "ec6665a5-bce9-41b5-98ca-d7a46e344e62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:11.615206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.615083 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec6665a5-bce9-41b5-98ca-d7a46e344e62-kube-api-access-wn4kc" pod "ec6665a5-bce9-41b5-98ca-d7a46e344e62" (UID: "ec6665a5-bce9-41b5-98ca-d7a46e344e62"). InnerVolumeSpecName "kube-api-access-wn4kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:11.615729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.615708 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-metrics-certs" pod "ec6665a5-bce9-41b5-98ca-d7a46e344e62" (UID: "ec6665a5-bce9-41b5-98ca-d7a46e344e62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:11.629210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.629187 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:11.714337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.714295 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn4kc\" (UniqueName: \"kubernetes.io/projected/ec6665a5-bce9-41b5-98ca-d7a46e344e62-kube-api-access-wn4kc\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:11.714337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.714333 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:11.714337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.714347 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec6665a5-bce9-41b5-98ca-d7a46e344e62-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:11.774573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.774490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:11.791183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.791154 4573 generic.go:356] "Generic (PLEG): container finished" podID="ec6665a5-bce9-41b5-98ca-d7a46e344e62" containerID="1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d" exitCode=0 Nov 01 16:33:11.791383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.791205 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" Nov 01 16:33:11.791383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.791233 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" event={"ID":"ec6665a5-bce9-41b5-98ca-d7a46e344e62","Type":"ContainerDied","Data":"1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d"} Nov 01 16:33:11.791383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.791296 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq" event={"ID":"ec6665a5-bce9-41b5-98ca-d7a46e344e62","Type":"ContainerDied","Data":"1c1028fc4944e5b0bfca74f96db43186d6ece3b008abae2d31ae87028281456c"} Nov 01 16:33:11.791383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.791328 4573 scope.go:122] "RemoveContainer" containerID="1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d" Nov 01 16:33:11.802656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.802637 4573 scope.go:122] "RemoveContainer" containerID="1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d" Nov 01 16:33:11.802950 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:11.802929 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d\": container with ID starting with 1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d not found: ID does not exist" containerID="1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d" Nov 01 16:33:11.803007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.802957 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d"} err="failed to get container status \"1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d\": rpc error: code = NotFound desc = could not find container \"1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d\": container with ID starting with 1fad4fbabba8be3ce25e6c9ce35251ebdc96b428d1c597fd839ef54d3326246d not found: ID does not exist" Nov 01 16:33:11.816431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.816407 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq"] Nov 01 16:33:11.820348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.820326 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-5ccd7b-77bbd66bc5-djsvq"] Nov 01 16:33:11.998482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:11.998447 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:12.458781 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:12.458732 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d10ba85-9fe1-4f4b-970b-825f667c1649" path="/var/lib/kubelet/pods/2d10ba85-9fe1-4f4b-970b-825f667c1649/volumes" Nov 01 16:33:12.459249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:12.459234 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec6665a5-bce9-41b5-98ca-d7a46e344e62" path="/var/lib/kubelet/pods/ec6665a5-bce9-41b5-98ca-d7a46e344e62/volumes" Nov 01 16:33:12.464028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:12.464005 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:13.142890 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:13.142850 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:13.705493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:13.705453 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:13.868774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:13.859704 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:14.020564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:14.020467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:14.034602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:14.034559 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q"] Nov 01 16:33:14.038083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:14.038061 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-9a482b-j955q"] Nov 01 16:33:14.115635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:14.115602 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:14.459321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:14.459283 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8af9bdb9-8281-425a-adab-1f716019b7cb" path="/var/lib/kubelet/pods/8af9bdb9-8281-425a-adab-1f716019b7cb/volumes" Nov 01 16:33:14.468935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:14.468894 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:14.825417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:14.825328 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:15.453993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.453961 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:15.459881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.459849 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" podUID="c5b94c21-751f-4484-9cfd-8100407cb40f" containerName="payload-processing" containerID="cri-o://c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa" gracePeriod=25 Nov 01 16:33:15.988155 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988120 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk"] Nov 01 16:33:15.988707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988692 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a431f4fc-4c91-49b0-8ee4-a7bff87f8580" containerName="revoke-keys" Nov 01 16:33:15.988785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988708 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a431f4fc-4c91-49b0-8ee4-a7bff87f8580" containerName="revoke-keys" Nov 01 16:33:15.988785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988765 4573 cpu_manager.go:414] "containerMap: removing container" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" Nov 01 16:33:15.988785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988770 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" Nov 01 16:33:15.988785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988779 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ec6665a5-bce9-41b5-98ca-d7a46e344e62" containerName="maas-api" Nov 01 16:33:15.988785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988786 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec6665a5-bce9-41b5-98ca-d7a46e344e62" containerName="maas-api" Nov 01 16:33:15.988960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988880 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" Nov 01 16:33:15.988960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988891 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a431f4fc-4c91-49b0-8ee4-a7bff87f8580" containerName="revoke-keys" Nov 01 16:33:15.988960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.988904 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec6665a5-bce9-41b5-98ca-d7a46e344e62" containerName="maas-api" Nov 01 16:33:15.993851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.993835 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:15.999167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.999142 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2"] Nov 01 16:33:15.999824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.999784 4573 cpu_manager.go:414] "containerMap: removing container" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" Nov 01 16:33:15.999824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:15.999808 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" Nov 01 16:33:16.000057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.000038 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a241e7-c6e7-4a85-80d4-906fbde4253d" containerName="istio-proxy" Nov 01 16:33:16.002346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.002322 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:33:16.002954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.002932 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:33:16.011923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.011899 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk"] Nov 01 16:33:16.012079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.011975 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2"] Nov 01 16:33:16.012079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.012076 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.108899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.108867 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.109073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.109009 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8v2\" (UniqueName: \"kubernetes.io/projected/6b0dc0f7-debd-4c72-a795-47b2f17144d5-kube-api-access-vj8v2\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.109073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.109064 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.109156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.109118 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zvvj\" (UniqueName: \"kubernetes.io/projected/bcb64a0d-5f25-47f4-8397-83fc89da9381-kube-api-access-2zvvj\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.109156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.109145 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.109288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.109258 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.210057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.210022 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.210263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.210122 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.210263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.210235 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8v2\" (UniqueName: \"kubernetes.io/projected/6b0dc0f7-debd-4c72-a795-47b2f17144d5-kube-api-access-vj8v2\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.210393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.210304 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.210393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.210350 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zvvj\" (UniqueName: \"kubernetes.io/projected/bcb64a0d-5f25-47f4-8397-83fc89da9381-kube-api-access-2zvvj\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.210393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.210385 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.210797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.210738 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.210941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.210922 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.212621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.212597 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.212716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.212665 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.218011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.217985 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zvvj\" (UniqueName: \"kubernetes.io/projected/bcb64a0d-5f25-47f4-8397-83fc89da9381-kube-api-access-2zvvj\") pod \"maas-api-revoke-sub-fdd98059-8x8b2\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.218011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.218006 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8v2\" (UniqueName: \"kubernetes.io/projected/6b0dc0f7-debd-4c72-a795-47b2f17144d5-kube-api-access-vj8v2\") pod \"maas-api-revoke-sub-4a11d637-m75zk\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.242780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.242692 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:16.325267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.325230 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:16.331724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.331699 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:16.617021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.616993 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk"] Nov 01 16:33:16.617420 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:16.617149 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0dc0f7_debd_4c72_a795_47b2f17144d5.slice/crio-95d250bba5c4efe19e40ba7b1f075d1789222528d134cc7457e4770bb9ddc26f WatchSource:0}: Error finding container 95d250bba5c4efe19e40ba7b1f075d1789222528d134cc7457e4770bb9ddc26f: Status 404 returned error can't find the container with id 95d250bba5c4efe19e40ba7b1f075d1789222528d134cc7457e4770bb9ddc26f Nov 01 16:33:16.635511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.635481 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2"] Nov 01 16:33:16.635990 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:16.635957 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb64a0d_5f25_47f4_8397_83fc89da9381.slice/crio-05d2aa54e17de79cb99a9d98acb76e83ca3f5a3d81e64a80f01614537c47f846 WatchSource:0}: Error finding container 05d2aa54e17de79cb99a9d98acb76e83ca3f5a3d81e64a80f01614537c47f846: Status 404 returned error can't find the container with id 05d2aa54e17de79cb99a9d98acb76e83ca3f5a3d81e64a80f01614537c47f846 Nov 01 16:33:16.836866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.836832 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" event={"ID":"6b0dc0f7-debd-4c72-a795-47b2f17144d5","Type":"ContainerStarted","Data":"95d250bba5c4efe19e40ba7b1f075d1789222528d134cc7457e4770bb9ddc26f"} Nov 01 16:33:16.838206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:16.838172 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" event={"ID":"bcb64a0d-5f25-47f4-8397-83fc89da9381","Type":"ContainerStarted","Data":"05d2aa54e17de79cb99a9d98acb76e83ca3f5a3d81e64a80f01614537c47f846"} Nov 01 16:33:17.067617 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:17.067579 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:17.182837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:17.182741 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" podUID="f82a8513-36f7-4799-a7c7-0c685a52631d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:17.823999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:17.823921 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:17.846994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:17.846964 4573 generic.go:356] "Generic (PLEG): container finished" podID="6b0dc0f7-debd-4c72-a795-47b2f17144d5" containerID="7596b45836069d3c939ddd0e819db86ae7c75f65c111194cc646ecdabac5eeee" exitCode=0 Nov 01 16:33:17.847160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:17.847039 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" event={"ID":"6b0dc0f7-debd-4c72-a795-47b2f17144d5","Type":"ContainerDied","Data":"7596b45836069d3c939ddd0e819db86ae7c75f65c111194cc646ecdabac5eeee"} Nov 01 16:33:17.848590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:17.848569 4573 generic.go:356] "Generic (PLEG): container finished" podID="bcb64a0d-5f25-47f4-8397-83fc89da9381" containerID="675b1e1f0a5d52b8586284be4a32bbb6a250460ed4301e91bc1ab34058d1e7aa" exitCode=0 Nov 01 16:33:17.848710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:17.848662 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" event={"ID":"bcb64a0d-5f25-47f4-8397-83fc89da9381","Type":"ContainerDied","Data":"675b1e1f0a5d52b8586284be4a32bbb6a250460ed4301e91bc1ab34058d1e7aa"} Nov 01 16:33:18.192598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:18.192559 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" podUID="fde1a22e-1a29-40fe-ae79-9b3f717fc392" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:18.568874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:18.568797 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:18.663183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:18.663152 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:18.887795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:18.886889 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:19.166000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.165972 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:19.169737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.169714 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:19.276446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.276415 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-openshift-ca\") pod \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " Nov 01 16:33:19.276714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.276460 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-xks-ca\") pod \"bcb64a0d-5f25-47f4-8397-83fc89da9381\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " Nov 01 16:33:19.276714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.276487 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb64a0d-5f25-47f4-8397-83fc89da9381-kube-api-access-2zvvj\" (UniqueName: \"kubernetes.io/projected/bcb64a0d-5f25-47f4-8397-83fc89da9381-kube-api-access-2zvvj\") pod \"bcb64a0d-5f25-47f4-8397-83fc89da9381\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " Nov 01 16:33:19.276714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.276562 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-openshift-ca\") pod \"bcb64a0d-5f25-47f4-8397-83fc89da9381\" (UID: \"bcb64a0d-5f25-47f4-8397-83fc89da9381\") " Nov 01 16:33:19.276714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.276669 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-xks-ca\") pod \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " Nov 01 16:33:19.276714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.276710 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b0dc0f7-debd-4c72-a795-47b2f17144d5-kube-api-access-vj8v2\" (UniqueName: \"kubernetes.io/projected/6b0dc0f7-debd-4c72-a795-47b2f17144d5-kube-api-access-vj8v2\") pod \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\" (UID: \"6b0dc0f7-debd-4c72-a795-47b2f17144d5\") " Nov 01 16:33:19.277029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.276902 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-openshift-ca" pod "6b0dc0f7-debd-4c72-a795-47b2f17144d5" (UID: "6b0dc0f7-debd-4c72-a795-47b2f17144d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:19.277433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.277266 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:19.277433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.277306 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-openshift-ca" pod "bcb64a0d-5f25-47f4-8397-83fc89da9381" (UID: "bcb64a0d-5f25-47f4-8397-83fc89da9381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:19.278984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.278963 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb64a0d-5f25-47f4-8397-83fc89da9381-kube-api-access-2zvvj" pod "bcb64a0d-5f25-47f4-8397-83fc89da9381" (UID: "bcb64a0d-5f25-47f4-8397-83fc89da9381"). InnerVolumeSpecName "kube-api-access-2zvvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:19.279104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.279040 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-xks-ca" pod "bcb64a0d-5f25-47f4-8397-83fc89da9381" (UID: "bcb64a0d-5f25-47f4-8397-83fc89da9381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:19.279254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.279233 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0dc0f7-debd-4c72-a795-47b2f17144d5-kube-api-access-vj8v2" pod "6b0dc0f7-debd-4c72-a795-47b2f17144d5" (UID: "6b0dc0f7-debd-4c72-a795-47b2f17144d5"). InnerVolumeSpecName "kube-api-access-vj8v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:19.279320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.279252 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-xks-ca" pod "6b0dc0f7-debd-4c72-a795-47b2f17144d5" (UID: "6b0dc0f7-debd-4c72-a795-47b2f17144d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:19.364418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.364375 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:19.379042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.378970 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:19.379042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.378997 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zvvj\" (UniqueName: \"kubernetes.io/projected/bcb64a0d-5f25-47f4-8397-83fc89da9381-kube-api-access-2zvvj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:19.379042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.379014 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bcb64a0d-5f25-47f4-8397-83fc89da9381-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:19.379042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.379024 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b0dc0f7-debd-4c72-a795-47b2f17144d5-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:19.379042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.379033 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8v2\" (UniqueName: \"kubernetes.io/projected/6b0dc0f7-debd-4c72-a795-47b2f17144d5-kube-api-access-vj8v2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:19.437181 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.437149 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4"] Nov 01 16:33:19.437786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.437746 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6b0dc0f7-debd-4c72-a795-47b2f17144d5" containerName="revoke-keys" Nov 01 16:33:19.437786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.437780 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b0dc0f7-debd-4c72-a795-47b2f17144d5" containerName="revoke-keys" Nov 01 16:33:19.437946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.437823 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bcb64a0d-5f25-47f4-8397-83fc89da9381" containerName="revoke-keys" Nov 01 16:33:19.437946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.437829 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb64a0d-5f25-47f4-8397-83fc89da9381" containerName="revoke-keys" Nov 01 16:33:19.437946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.437940 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b0dc0f7-debd-4c72-a795-47b2f17144d5" containerName="revoke-keys" Nov 01 16:33:19.438065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.437963 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb64a0d-5f25-47f4-8397-83fc89da9381" containerName="revoke-keys" Nov 01 16:33:19.441690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.441662 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.448379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.448356 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4"] Nov 01 16:33:19.479889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.479854 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p62w\" (UniqueName: \"kubernetes.io/projected/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-kube-api-access-8p62w\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.479889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.479976 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.480370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.480119 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.581524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.581487 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.581741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.581707 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.581908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.581887 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p62w\" (UniqueName: \"kubernetes.io/projected/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-kube-api-access-8p62w\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.582650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.582622 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.584168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.584145 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.590627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.590604 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p62w\" (UniqueName: \"kubernetes.io/projected/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-kube-api-access-8p62w\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.766435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.766338 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:19.875280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.875247 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" Nov 01 16:33:19.875487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.875286 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2" event={"ID":"bcb64a0d-5f25-47f4-8397-83fc89da9381","Type":"ContainerDied","Data":"05d2aa54e17de79cb99a9d98acb76e83ca3f5a3d81e64a80f01614537c47f846"} Nov 01 16:33:19.875487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.875347 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05d2aa54e17de79cb99a9d98acb76e83ca3f5a3d81e64a80f01614537c47f846" Nov 01 16:33:19.877278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.877249 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" event={"ID":"6b0dc0f7-debd-4c72-a795-47b2f17144d5","Type":"ContainerDied","Data":"95d250bba5c4efe19e40ba7b1f075d1789222528d134cc7457e4770bb9ddc26f"} Nov 01 16:33:19.877278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.877273 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk" Nov 01 16:33:19.877505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:19.877299 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95d250bba5c4efe19e40ba7b1f075d1789222528d134cc7457e4770bb9ddc26f" Nov 01 16:33:20.043686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.043661 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4"] Nov 01 16:33:20.045798 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:20.045739 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdfe623c_be76_460e_a5f4_8d61b94e3fd8.slice/crio-9967a046781cb4b2d0914bd7349110b17f2241ffa799ab3f6bafacfde13c667d WatchSource:0}: Error finding container 9967a046781cb4b2d0914bd7349110b17f2241ffa799ab3f6bafacfde13c667d: Status 404 returned error can't find the container with id 9967a046781cb4b2d0914bd7349110b17f2241ffa799ab3f6bafacfde13c667d Nov 01 16:33:20.256575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.256531 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" containerID="cri-o://a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c" gracePeriod=45 Nov 01 16:33:20.261714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.261551 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:20.291653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.291620 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" containerID="cri-o://9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587" gracePeriod=45 Nov 01 16:33:20.420740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.420695 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": dial tcp 10.133.0.54:8000: connect: connection refused" Nov 01 16:33:20.571381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.571348 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": dial tcp 10.133.0.57:8000: connect: connection refused" Nov 01 16:33:20.681644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.681613 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:33:20.684942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.684918 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:33:20.800495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800464 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-model-cache\") pod \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " Nov 01 16:33:20.800706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800530 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tls-certs\") pod \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " Nov 01 16:33:20.800706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800586 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-kube-api-access-5rjn5\" (UniqueName: \"kubernetes.io/projected/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-kube-api-access-5rjn5\") pod \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " Nov 01 16:33:20.800706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800630 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tls-certs\") pod \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " Nov 01 16:33:20.800706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800672 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-kube-api-access-8tgvg\" (UniqueName: \"kubernetes.io/projected/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-kube-api-access-8tgvg\") pod \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " Nov 01 16:33:20.800990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800828 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-model-cache" pod "47dfa29a-24f6-40a0-8505-be7c5f4a36c3" (UID: "47dfa29a-24f6-40a0-8505-be7c5f4a36c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.800990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800841 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-dshm\") pod \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " Nov 01 16:33:20.800990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800914 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-model-cache\") pod \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " Nov 01 16:33:20.801166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.800990 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tmp-dir\") pod \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " Nov 01 16:33:20.801166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.801056 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tmp-dir\") pod \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " Nov 01 16:33:20.801166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.801088 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-home\") pod \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\" (UID: \"0f3d46b3-cd7c-4340-a37e-3571eb7b5892\") " Nov 01 16:33:20.801166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.801128 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-dshm\") pod \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " Nov 01 16:33:20.801166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.801162 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-home\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-home\") pod \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\" (UID: \"47dfa29a-24f6-40a0-8505-be7c5f4a36c3\") " Nov 01 16:33:20.802312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.801767 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-home" pod "0f3d46b3-cd7c-4340-a37e-3571eb7b5892" (UID: "0f3d46b3-cd7c-4340-a37e-3571eb7b5892"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.802312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802057 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tmp-dir" pod "0f3d46b3-cd7c-4340-a37e-3571eb7b5892" (UID: "0f3d46b3-cd7c-4340-a37e-3571eb7b5892"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.802312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802100 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-model-cache" pod "0f3d46b3-cd7c-4340-a37e-3571eb7b5892" (UID: "0f3d46b3-cd7c-4340-a37e-3571eb7b5892"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.802312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802278 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-home" pod "47dfa29a-24f6-40a0-8505-be7c5f4a36c3" (UID: "47dfa29a-24f6-40a0-8505-be7c5f4a36c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.802630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802390 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tmp-dir" pod "47dfa29a-24f6-40a0-8505-be7c5f4a36c3" (UID: "47dfa29a-24f6-40a0-8505-be7c5f4a36c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.802630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802414 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.802630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802432 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.802630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802445 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.802630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802458 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.802630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.802470 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.803769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.803721 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tls-certs" pod "0f3d46b3-cd7c-4340-a37e-3571eb7b5892" (UID: "0f3d46b3-cd7c-4340-a37e-3571eb7b5892"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:20.804221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.804197 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tls-certs" pod "47dfa29a-24f6-40a0-8505-be7c5f4a36c3" (UID: "47dfa29a-24f6-40a0-8505-be7c5f4a36c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:20.807113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.807076 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-dshm" pod "47dfa29a-24f6-40a0-8505-be7c5f4a36c3" (UID: "47dfa29a-24f6-40a0-8505-be7c5f4a36c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.807113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.807097 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-dshm" pod "0f3d46b3-cd7c-4340-a37e-3571eb7b5892" (UID: "0f3d46b3-cd7c-4340-a37e-3571eb7b5892"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:20.807297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.807137 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-kube-api-access-5rjn5" pod "0f3d46b3-cd7c-4340-a37e-3571eb7b5892" (UID: "0f3d46b3-cd7c-4340-a37e-3571eb7b5892"). InnerVolumeSpecName "kube-api-access-5rjn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:20.808533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.808514 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-kube-api-access-8tgvg" pod "47dfa29a-24f6-40a0-8505-be7c5f4a36c3" (UID: "47dfa29a-24f6-40a0-8505-be7c5f4a36c3"). InnerVolumeSpecName "kube-api-access-8tgvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:20.885915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.885880 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" event={"ID":"cdfe623c-be76-460e-a5f4-8d61b94e3fd8","Type":"ContainerStarted","Data":"64e6beffd4ade3b0aa5b952878a63fb4ca70dd09f231e8e424652755b59ea125"} Nov 01 16:33:20.886100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.885947 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" event={"ID":"cdfe623c-be76-460e-a5f4-8d61b94e3fd8","Type":"ContainerStarted","Data":"9967a046781cb4b2d0914bd7349110b17f2241ffa799ab3f6bafacfde13c667d"} Nov 01 16:33:20.887270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.887240 4573 generic.go:356] "Generic (PLEG): container finished" podID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerID="9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587" exitCode=0 Nov 01 16:33:20.887402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.887279 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" Nov 01 16:33:20.887402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.887287 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" event={"ID":"0f3d46b3-cd7c-4340-a37e-3571eb7b5892","Type":"ContainerDied","Data":"9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587"} Nov 01 16:33:20.887402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.887326 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9" event={"ID":"0f3d46b3-cd7c-4340-a37e-3571eb7b5892","Type":"ContainerDied","Data":"5883ea9816e05df2976a32048623b26c891d44bc040c5d461bc2a35e3a7f7c93"} Nov 01 16:33:20.887402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.887345 4573 scope.go:122] "RemoveContainer" containerID="9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587" Nov 01 16:33:20.889501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.888929 4573 generic.go:356] "Generic (PLEG): container finished" podID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerID="a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c" exitCode=0 Nov 01 16:33:20.889501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.888969 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" Nov 01 16:33:20.889501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.888962 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" event={"ID":"47dfa29a-24f6-40a0-8505-be7c5f4a36c3","Type":"ContainerDied","Data":"a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c"} Nov 01 16:33:20.889501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.889067 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9" event={"ID":"47dfa29a-24f6-40a0-8505-be7c5f4a36c3","Type":"ContainerDied","Data":"4a13dd87e4daf887863991abd71f6cc6fe8dfb7f3c86f7e570ffd5a0efaec1b5"} Nov 01 16:33:20.903906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.903882 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.904017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.903909 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tgvg\" (UniqueName: \"kubernetes.io/projected/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-kube-api-access-8tgvg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.904017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.903924 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.904017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.903935 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.904017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.903942 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.904017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.903950 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47dfa29a-24f6-40a0-8505-be7c5f4a36c3-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.904017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.903959 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rjn5\" (UniqueName: \"kubernetes.io/projected/0f3d46b3-cd7c-4340-a37e-3571eb7b5892-kube-api-access-5rjn5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:20.904715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.904692 4573 scope.go:122] "RemoveContainer" containerID="9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587" Nov 01 16:33:20.905086 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:20.905058 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587\": container with ID starting with 9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587 not found: ID does not exist" containerID="9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587" Nov 01 16:33:20.905189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.905097 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587"} err="failed to get container status \"9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587\": rpc error: code = NotFound desc = could not find container \"9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587\": container with ID starting with 9832a1edb5c9ddbac89e95590c1967ff183e1bac2b5bbc5b85af85c681afc587 not found: ID does not exist" Nov 01 16:33:20.905189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.905120 4573 scope.go:122] "RemoveContainer" containerID="a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c" Nov 01 16:33:20.905429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.905223 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" podStartSLOduration=1.9052063769999998 podStartE2EDuration="1.905206377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:20.902611776 +0000 UTC m=+2738.605777713" watchObservedRunningTime="2026-11-01 16:33:20.905206377 +0000 UTC m=+2738.608372314" Nov 01 16:33:20.916323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.916305 4573 scope.go:122] "RemoveContainer" containerID="a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c" Nov 01 16:33:20.916593 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:20.916569 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c\": container with ID starting with a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c not found: ID does not exist" containerID="a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c" Nov 01 16:33:20.916654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.916605 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c"} err="failed to get container status \"a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c\": rpc error: code = NotFound desc = could not find container \"a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c\": container with ID starting with a73d321f2e2b39aa28ccfcf21f9adbc947b03d9a145dd2ed559afbeb35cbc48c not found: ID does not exist" Nov 01 16:33:20.926493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.926462 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9"] Nov 01 16:33:20.934566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.934541 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/simulator-w3-bd831a-kserve-676dc75c6f-4w6b9"] Nov 01 16:33:20.945375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.945349 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9"] Nov 01 16:33:20.951034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.951009 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bd831a/premium-w3-bd831a-kserve-7445b5dc84-nc8w9"] Nov 01 16:33:20.989400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:20.989363 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:21.765794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:21.765742 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:21.899804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:21.899738 4573 generic.go:356] "Generic (PLEG): container finished" podID="cdfe623c-be76-460e-a5f4-8d61b94e3fd8" containerID="64e6beffd4ade3b0aa5b952878a63fb4ca70dd09f231e8e424652755b59ea125" exitCode=0 Nov 01 16:33:21.899804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:21.899795 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" event={"ID":"cdfe623c-be76-460e-a5f4-8d61b94e3fd8","Type":"ContainerDied","Data":"64e6beffd4ade3b0aa5b952878a63fb4ca70dd09f231e8e424652755b59ea125"} Nov 01 16:33:22.107439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.107411 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:22.206165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.206133 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:22.390694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.390606 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt"] Nov 01 16:33:22.390934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.390903 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" containerID="cri-o://ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71" gracePeriod=30 Nov 01 16:33:22.404558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.402296 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747"] Nov 01 16:33:22.404558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.402520 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" containerID="cri-o://6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b" gracePeriod=30 Nov 01 16:33:22.428232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.428202 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524"] Nov 01 16:33:22.428452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.428428 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" containerID="cri-o://db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6" gracePeriod=30 Nov 01 16:33:22.443363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.443327 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:22.461883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.461856 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" path="/var/lib/kubelet/pods/0f3d46b3-cd7c-4340-a37e-3571eb7b5892/volumes" Nov 01 16:33:22.463306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.463264 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" path="/var/lib/kubelet/pods/47dfa29a-24f6-40a0-8505-be7c5f4a36c3/volumes" Nov 01 16:33:22.737007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.736798 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:22.883582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.883548 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h"] Nov 01 16:33:22.884255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.884236 4573 cpu_manager.go:414] "containerMap: removing container" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" Nov 01 16:33:22.884358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.884256 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" Nov 01 16:33:22.884358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.884280 4573 cpu_manager.go:414] "containerMap: removing container" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" Nov 01 16:33:22.884358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.884288 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" Nov 01 16:33:22.884529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.884398 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3d46b3-cd7c-4340-a37e-3571eb7b5892" containerName="main" Nov 01 16:33:22.884529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.884409 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="47dfa29a-24f6-40a0-8505-be7c5f4a36c3" containerName="main" Nov 01 16:33:22.889107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.889089 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:22.895256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:22.895229 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h"] Nov 01 16:33:23.040592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.040553 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.040842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.040814 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p725q\" (UniqueName: \"kubernetes.io/projected/3698bc5b-a020-4f29-8164-00d35322a558-kube-api-access-p725q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.040998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.040980 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.052082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.052057 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:23.141870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.141843 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-openshift-ca\") pod \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " Nov 01 16:33:23.142019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.141875 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-xks-ca\") pod \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " Nov 01 16:33:23.142019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.141942 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-kube-api-access-8p62w\" (UniqueName: \"kubernetes.io/projected/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-kube-api-access-8p62w\") pod \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\" (UID: \"cdfe623c-be76-460e-a5f4-8d61b94e3fd8\") " Nov 01 16:33:23.142199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.142177 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-openshift-ca" pod "cdfe623c-be76-460e-a5f4-8d61b94e3fd8" (UID: "cdfe623c-be76-460e-a5f4-8d61b94e3fd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:23.142338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.142305 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p725q\" (UniqueName: \"kubernetes.io/projected/3698bc5b-a020-4f29-8164-00d35322a558-kube-api-access-p725q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.142480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.142416 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.142480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.142470 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.142623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.142600 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:23.143292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.143267 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.144190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.144172 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-xks-ca" pod "cdfe623c-be76-460e-a5f4-8d61b94e3fd8" (UID: "cdfe623c-be76-460e-a5f4-8d61b94e3fd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:23.144257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.144197 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-kube-api-access-8p62w" pod "cdfe623c-be76-460e-a5f4-8d61b94e3fd8" (UID: "cdfe623c-be76-460e-a5f4-8d61b94e3fd8"). InnerVolumeSpecName "kube-api-access-8p62w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:23.144909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.144893 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.149851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.149830 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p725q\" (UniqueName: \"kubernetes.io/projected/3698bc5b-a020-4f29-8164-00d35322a558-kube-api-access-p725q\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.207277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.207244 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:23.243523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.243485 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p62w\" (UniqueName: \"kubernetes.io/projected/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-kube-api-access-8p62w\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:23.243523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.243516 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdfe623c-be76-460e-a5f4-8d61b94e3fd8-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:23.344087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.343991 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" podUID="c5b94c21-751f-4484-9cfd-8100407cb40f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:23.487835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.487801 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h"] Nov 01 16:33:23.488210 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:23.488187 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3698bc5b_a020_4f29_8164_00d35322a558.slice/crio-62e04d67ef1f758255861c509fcc1d550d749af19dc9ac3b872656953bb5662c WatchSource:0}: Error finding container 62e04d67ef1f758255861c509fcc1d550d749af19dc9ac3b872656953bb5662c: Status 404 returned error can't find the container with id 62e04d67ef1f758255861c509fcc1d550d749af19dc9ac3b872656953bb5662c Nov 01 16:33:23.661613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.661538 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:23.916885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.916809 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" Nov 01 16:33:23.917311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.916807 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4" event={"ID":"cdfe623c-be76-460e-a5f4-8d61b94e3fd8","Type":"ContainerDied","Data":"9967a046781cb4b2d0914bd7349110b17f2241ffa799ab3f6bafacfde13c667d"} Nov 01 16:33:23.917311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.916943 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9967a046781cb4b2d0914bd7349110b17f2241ffa799ab3f6bafacfde13c667d" Nov 01 16:33:23.918642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.918619 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" event={"ID":"3698bc5b-a020-4f29-8164-00d35322a558","Type":"ContainerStarted","Data":"ebe50d6c5cc80b64b6018a2a623a3cf2ae5b6d942f12dd095d40957fefcc61aa"} Nov 01 16:33:23.918819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.918660 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" event={"ID":"3698bc5b-a020-4f29-8164-00d35322a558","Type":"ContainerStarted","Data":"62e04d67ef1f758255861c509fcc1d550d749af19dc9ac3b872656953bb5662c"} Nov 01 16:33:23.943143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:23.943099 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" podStartSLOduration=1.9430870919999998 podStartE2EDuration="1.943087092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:23.938521927 +0000 UTC m=+2741.641687866" watchObservedRunningTime="2026-11-01 16:33:23.943087092 +0000 UTC m=+2741.646253041" Nov 01 16:33:24.364263 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:24.364228 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:33:24.465469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:24.465440 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:24.927239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:24.927209 4573 generic.go:356] "Generic (PLEG): container finished" podID="3698bc5b-a020-4f29-8164-00d35322a558" containerID="ebe50d6c5cc80b64b6018a2a623a3cf2ae5b6d942f12dd095d40957fefcc61aa" exitCode=0 Nov 01 16:33:24.927658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:24.927294 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" event={"ID":"3698bc5b-a020-4f29-8164-00d35322a558","Type":"ContainerDied","Data":"ebe50d6c5cc80b64b6018a2a623a3cf2ae5b6d942f12dd095d40957fefcc61aa"} Nov 01 16:33:25.161685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:25.161644 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:25.284498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:25.284403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:25.670475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:25.670429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:25.808540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:25.808491 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:26.085681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.085653 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:26.088963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.088941 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:26.203376 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.203297 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3698bc5b-a020-4f29-8164-00d35322a558-kube-api-access-p725q\" (UniqueName: \"kubernetes.io/projected/3698bc5b-a020-4f29-8164-00d35322a558-kube-api-access-p725q\") pod \"3698bc5b-a020-4f29-8164-00d35322a558\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " Nov 01 16:33:26.203527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.203413 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-openshift-ca\") pod \"3698bc5b-a020-4f29-8164-00d35322a558\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " Nov 01 16:33:26.203626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.203608 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-xks-ca\") pod \"3698bc5b-a020-4f29-8164-00d35322a558\" (UID: \"3698bc5b-a020-4f29-8164-00d35322a558\") " Nov 01 16:33:26.203875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.203835 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-openshift-ca" pod "3698bc5b-a020-4f29-8164-00d35322a558" (UID: "3698bc5b-a020-4f29-8164-00d35322a558"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:26.204290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.204262 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:26.205683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.205644 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-xks-ca" pod "3698bc5b-a020-4f29-8164-00d35322a558" (UID: "3698bc5b-a020-4f29-8164-00d35322a558"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:26.205839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.205723 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3698bc5b-a020-4f29-8164-00d35322a558-kube-api-access-p725q" pod "3698bc5b-a020-4f29-8164-00d35322a558" (UID: "3698bc5b-a020-4f29-8164-00d35322a558"). InnerVolumeSpecName "kube-api-access-p725q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:26.306182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.306142 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3698bc5b-a020-4f29-8164-00d35322a558-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:26.306182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.306180 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p725q\" (UniqueName: \"kubernetes.io/projected/3698bc5b-a020-4f29-8164-00d35322a558-kube-api-access-p725q\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:26.850098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.850009 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:26.945598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.945569 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" Nov 01 16:33:26.945784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.945561 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h" event={"ID":"3698bc5b-a020-4f29-8164-00d35322a558","Type":"ContainerDied","Data":"62e04d67ef1f758255861c509fcc1d550d749af19dc9ac3b872656953bb5662c"} Nov 01 16:33:26.945784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:26.945704 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62e04d67ef1f758255861c509fcc1d550d749af19dc9ac3b872656953bb5662c" Nov 01 16:33:27.182828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.182744 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" podUID="f82a8513-36f7-4799-a7c7-0c685a52631d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:27.539114 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:27.539080 4573 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e6721a2_bcda_4eab_a726_90812e0dfb06.slice/crio-6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b.scope\": RecentStats: unable to find data in memory cache]" Nov 01 16:33:27.540815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.540729 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:27.730857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.730004 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:33:27.771863 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.771836 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:33:27.793219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.793098 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833004 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-envoy\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833122 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-socket\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833179 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-data\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833225 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e6721a2-bcda-4eab-a726-90812e0dfb06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6721a2-bcda-4eab-a726-90812e0dfb06-istiod-ca-cert\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833303 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-token\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-token\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833384 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-kube-api-access-nlvxk\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-kube-api-access-nlvxk\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833447 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-podinfo\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833481 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-certs\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.833627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833522 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-credential-socket\") pod \"7e6721a2-bcda-4eab-a726-90812e0dfb06\" (UID: \"7e6721a2-bcda-4eab-a726-90812e0dfb06\") " Nov 01 16:33:27.834306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.833903 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e6721a2-bcda-4eab-a726-90812e0dfb06-istiod-ca-cert" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:27.834306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.834177 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-credential-socket" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.834306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.834285 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e6721a2-bcda-4eab-a726-90812e0dfb06-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.834306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.834305 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.834548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.834376 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-socket" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.834614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.834577 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-data" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.835220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.835193 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-certs" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.837079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.837055 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-podinfo" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:27.837223 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.837204 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-envoy" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.837678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.837647 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-kube-api-access-nlvxk" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "kube-api-access-nlvxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.838608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.838583 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-token" pod "7e6721a2-bcda-4eab-a726-90812e0dfb06" (UID: "7e6721a2-bcda-4eab-a726-90812e0dfb06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.934813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.934783 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-envoy\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.934941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.934842 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-kube-api-access-bgbgb\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-kube-api-access-bgbgb\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.934941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.934897 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-socket\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.934941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.934931 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istiod-ca-cert\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.935116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.934961 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-certs\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.935116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935007 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-kube-api-access-pghxk\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-kube-api-access-pghxk\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.935116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935037 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-token\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-token\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.935116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935084 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-socket\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.935333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935122 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istiod-ca-cert\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.935333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935179 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-data\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.935333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935223 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-data\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.935333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935260 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-certs" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.935333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935277 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-podinfo\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.935333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935298 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-socket" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.935333 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935322 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-envoy\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.935708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935343 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istiod-ca-cert" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:27.935708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935352 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-token\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-token\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.935708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935411 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-credential-socket\") pod \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\" (UID: \"1c3ec6a5-f442-4335-b93c-b6bcea7de99f\") " Nov 01 16:33:27.935708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935543 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-data" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.935708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935589 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istiod-ca-cert" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:27.935708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935596 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-certs\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.936085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935813 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-data" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.936085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935862 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-credential-socket\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.936085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935902 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-certs" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.936085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.935944 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-podinfo\") pod \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\" (UID: \"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0\") " Nov 01 16:33:27.936327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936127 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-socket" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.936327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936166 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-credential-socket" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.936327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936228 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-credential-socket" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.936534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936512 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936538 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936553 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936568 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936582 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936597 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936612 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936626 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936640 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936654 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936671 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlvxk\" (UniqueName: \"kubernetes.io/projected/7e6721a2-bcda-4eab-a726-90812e0dfb06-kube-api-access-nlvxk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936687 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936700 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936715 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936730 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936745 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e6721a2-bcda-4eab-a726-90812e0dfb06-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.936933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.936778 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:27.937565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.937327 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-envoy" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.937942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.937895 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-kube-api-access-bgbgb" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "kube-api-access-bgbgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.938093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.937992 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-envoy" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:27.938349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.938327 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-token" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.938581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.938558 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-podinfo" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:27.938692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.938597 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-token" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.938875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.938856 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-kube-api-access-pghxk" pod "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" (UID: "12add1b5-31ff-4c5c-b33b-cf8ffd361bf0"). InnerVolumeSpecName "kube-api-access-pghxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:27.939397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.939380 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-podinfo" pod "1c3ec6a5-f442-4335-b93c-b6bcea7de99f" (UID: "1c3ec6a5-f442-4335-b93c-b6bcea7de99f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:27.955122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.955095 4573 generic.go:356] "Generic (PLEG): container finished" podID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerID="6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b" exitCode=0 Nov 01 16:33:27.955212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.955193 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" Nov 01 16:33:27.955261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.955240 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" event={"ID":"7e6721a2-bcda-4eab-a726-90812e0dfb06","Type":"ContainerDied","Data":"6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b"} Nov 01 16:33:27.955311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.955297 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747" event={"ID":"7e6721a2-bcda-4eab-a726-90812e0dfb06","Type":"ContainerDied","Data":"08c7bc1a06c06a364920d0ef795a143714730eed6fd958d4858b46f31af5acb5"} Nov 01 16:33:27.955366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.955327 4573 scope.go:122] "RemoveContainer" containerID="6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b" Nov 01 16:33:27.956814 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.956789 4573 generic.go:356] "Generic (PLEG): container finished" podID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerID="ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71" exitCode=0 Nov 01 16:33:27.956919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.956817 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" Nov 01 16:33:27.956919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.956866 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" event={"ID":"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0","Type":"ContainerDied","Data":"ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71"} Nov 01 16:33:27.956919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.956915 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt" event={"ID":"12add1b5-31ff-4c5c-b33b-cf8ffd361bf0","Type":"ContainerDied","Data":"d0035f70598c4413a5c55c67cb915e9a29e8f373c8531bea309eb65cde854343"} Nov 01 16:33:27.958423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.958405 4573 generic.go:356] "Generic (PLEG): container finished" podID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerID="db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6" exitCode=0 Nov 01 16:33:27.958520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.958452 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" Nov 01 16:33:27.958579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.958535 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" event={"ID":"1c3ec6a5-f442-4335-b93c-b6bcea7de99f","Type":"ContainerDied","Data":"db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6"} Nov 01 16:33:27.958631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.958583 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524" event={"ID":"1c3ec6a5-f442-4335-b93c-b6bcea7de99f","Type":"ContainerDied","Data":"6afdc96535e7d42a0889208ef6bee4ce6426bf0631d428c5a7044b6ed284fef0"} Nov 01 16:33:27.975531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.975504 4573 scope.go:122] "RemoveContainer" containerID="6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b" Nov 01 16:33:27.975819 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:27.975789 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b\": container with ID starting with 6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b not found: ID does not exist" containerID="6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b" Nov 01 16:33:27.975915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.975824 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b"} err="failed to get container status \"6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b\": rpc error: code = NotFound desc = could not find container \"6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b\": container with ID starting with 6877a0afd2c2e8322bc22c44dd0cd5864a1d1189daa81fb87f3d4e98ab866c1b not found: ID does not exist" Nov 01 16:33:27.975915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.975843 4573 scope.go:122] "RemoveContainer" containerID="ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71" Nov 01 16:33:27.996311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.993271 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747"] Nov 01 16:33:27.996311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.994220 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-jn747"] Nov 01 16:33:27.999354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.999335 4573 scope.go:122] "RemoveContainer" containerID="ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71" Nov 01 16:33:27.999811 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:27.999744 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71\": container with ID starting with ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71 not found: ID does not exist" containerID="ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71" Nov 01 16:33:27.999903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.999821 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71"} err="failed to get container status \"ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71\": rpc error: code = NotFound desc = could not find container \"ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71\": container with ID starting with ccfe03876f023e90e4eb5af678f3457685aa6eb756a0f1a392fdfda8aa3edd71 not found: ID does not exist" Nov 01 16:33:27.999903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:27.999840 4573 scope.go:122] "RemoveContainer" containerID="db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6" Nov 01 16:33:28.010690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.010668 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt"] Nov 01 16:33:28.015878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.015854 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-2f4vt"] Nov 01 16:33:28.018759 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.018728 4573 scope.go:122] "RemoveContainer" containerID="db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6" Nov 01 16:33:28.019174 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:28.019153 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6\": container with ID starting with db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6 not found: ID does not exist" containerID="db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6" Nov 01 16:33:28.019300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.019183 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6"} err="failed to get container status \"db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6\": rpc error: code = NotFound desc = could not find container \"db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6\": container with ID starting with db4208ecfccd234f59e55c5022cf8078fdc4b7d817ade479af2d18fd71f886b6 not found: ID does not exist" Nov 01 16:33:28.028897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.028880 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524"] Nov 01 16:33:28.033804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.033782 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-9a482b-openshift-default-799986d6f9-cr524"] Nov 01 16:33:28.041311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.041173 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.041202 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgbgb\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-kube-api-access-bgbgb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.041219 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pghxk\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-kube-api-access-pghxk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.041235 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.041252 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.041265 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.041278 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3ec6a5-f442-4335-b93c-b6bcea7de99f-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.041311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.041291 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.122466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.122436 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:33:28.141885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.141851 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f82a8513-36f7-4799-a7c7-0c685a52631d-kube-api-access-m76rj\" (UniqueName: \"kubernetes.io/projected/f82a8513-36f7-4799-a7c7-0c685a52631d-kube-api-access-m76rj\") pod \"f82a8513-36f7-4799-a7c7-0c685a52631d\" (UID: \"f82a8513-36f7-4799-a7c7-0c685a52631d\") " Nov 01 16:33:28.141885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.141929 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f82a8513-36f7-4799-a7c7-0c685a52631d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a8513-36f7-4799-a7c7-0c685a52631d-plugins-config-volume\") pod \"f82a8513-36f7-4799-a7c7-0c685a52631d\" (UID: \"f82a8513-36f7-4799-a7c7-0c685a52631d\") " Nov 01 16:33:28.142823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.142789 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f82a8513-36f7-4799-a7c7-0c685a52631d-plugins-config-volume" pod "f82a8513-36f7-4799-a7c7-0c685a52631d" (UID: "f82a8513-36f7-4799-a7c7-0c685a52631d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:28.145292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.145265 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82a8513-36f7-4799-a7c7-0c685a52631d-kube-api-access-m76rj" pod "f82a8513-36f7-4799-a7c7-0c685a52631d" (UID: "f82a8513-36f7-4799-a7c7-0c685a52631d"). InnerVolumeSpecName "kube-api-access-m76rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:28.166338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.166313 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:33:28.243367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.243333 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fde1a22e-1a29-40fe-ae79-9b3f717fc392-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1a22e-1a29-40fe-ae79-9b3f717fc392-plugins-config-volume\") pod \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\" (UID: \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\") " Nov 01 16:33:28.243854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.243412 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fde1a22e-1a29-40fe-ae79-9b3f717fc392-kube-api-access-ztrj9\" (UniqueName: \"kubernetes.io/projected/fde1a22e-1a29-40fe-ae79-9b3f717fc392-kube-api-access-ztrj9\") pod \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\" (UID: \"fde1a22e-1a29-40fe-ae79-9b3f717fc392\") " Nov 01 16:33:28.243854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.243665 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde1a22e-1a29-40fe-ae79-9b3f717fc392-plugins-config-volume" pod "fde1a22e-1a29-40fe-ae79-9b3f717fc392" (UID: "fde1a22e-1a29-40fe-ae79-9b3f717fc392"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:28.244072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.244048 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fde1a22e-1a29-40fe-ae79-9b3f717fc392-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.244140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.244078 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m76rj\" (UniqueName: \"kubernetes.io/projected/f82a8513-36f7-4799-a7c7-0c685a52631d-kube-api-access-m76rj\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.244140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.244093 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f82a8513-36f7-4799-a7c7-0c685a52631d-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.245580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.245556 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde1a22e-1a29-40fe-ae79-9b3f717fc392-kube-api-access-ztrj9" pod "fde1a22e-1a29-40fe-ae79-9b3f717fc392" (UID: "fde1a22e-1a29-40fe-ae79-9b3f717fc392"). InnerVolumeSpecName "kube-api-access-ztrj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:28.345825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.345689 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztrj9\" (UniqueName: \"kubernetes.io/projected/fde1a22e-1a29-40fe-ae79-9b3f717fc392-kube-api-access-ztrj9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:28.350383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.350349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:28.465305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.465269 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" path="/var/lib/kubelet/pods/12add1b5-31ff-4c5c-b33b-cf8ffd361bf0/volumes" Nov 01 16:33:28.465738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.465724 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" path="/var/lib/kubelet/pods/1c3ec6a5-f442-4335-b93c-b6bcea7de99f/volumes" Nov 01 16:33:28.466433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.466416 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" path="/var/lib/kubelet/pods/7e6721a2-bcda-4eab-a726-90812e0dfb06/volumes" Nov 01 16:33:28.788878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.788841 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:28.888055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.888010 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:28.969778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.969732 4573 generic.go:356] "Generic (PLEG): container finished" podID="fde1a22e-1a29-40fe-ae79-9b3f717fc392" containerID="5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e" exitCode=0 Nov 01 16:33:28.969993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.969814 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" Nov 01 16:33:28.969993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.969813 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" event={"ID":"fde1a22e-1a29-40fe-ae79-9b3f717fc392","Type":"ContainerDied","Data":"5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e"} Nov 01 16:33:28.969993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.969922 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs" event={"ID":"fde1a22e-1a29-40fe-ae79-9b3f717fc392","Type":"ContainerDied","Data":"f5e9ebe1421bcd233bbb53ec3b9dfeff1a38d189f50d02710df5f1e1715dc7aa"} Nov 01 16:33:28.969993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.969948 4573 scope.go:122] "RemoveContainer" containerID="5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e" Nov 01 16:33:28.972872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.972849 4573 generic.go:356] "Generic (PLEG): container finished" podID="f82a8513-36f7-4799-a7c7-0c685a52631d" containerID="465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67" exitCode=0 Nov 01 16:33:28.973000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.972876 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" event={"ID":"f82a8513-36f7-4799-a7c7-0c685a52631d","Type":"ContainerDied","Data":"465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67"} Nov 01 16:33:28.973000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.972910 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" Nov 01 16:33:28.973000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.972926 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4" event={"ID":"f82a8513-36f7-4799-a7c7-0c685a52631d","Type":"ContainerDied","Data":"ad299db0f807c969201a09a9adcba8b5a5758e8b4ff6c4b4d914f71362d9f5d6"} Nov 01 16:33:28.980606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.980588 4573 scope.go:122] "RemoveContainer" containerID="5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e" Nov 01 16:33:28.980969 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:28.980950 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e\": container with ID starting with 5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e not found: ID does not exist" containerID="5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e" Nov 01 16:33:28.981024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.980979 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e"} err="failed to get container status \"5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e\": rpc error: code = NotFound desc = could not find container \"5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e\": container with ID starting with 5d760541e66f810848c2bac287be820e5cf8dc62fdfe8c9d56fae3b733cbe76e not found: ID does not exist" Nov 01 16:33:28.981024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.980995 4573 scope.go:122] "RemoveContainer" containerID="465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67" Nov 01 16:33:28.990938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.990920 4573 scope.go:122] "RemoveContainer" containerID="465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67" Nov 01 16:33:28.991216 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:28.991197 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67\": container with ID starting with 465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67 not found: ID does not exist" containerID="465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67" Nov 01 16:33:28.991278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.991223 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67"} err="failed to get container status \"465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67\": rpc error: code = NotFound desc = could not find container \"465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67\": container with ID starting with 465de4db9dd705d2687af1b54122d03097d987146820f00371b6250ad44b3d67 not found: ID does not exist" Nov 01 16:33:28.991459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:28.991440 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs"] Nov 01 16:33:29.015263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.015227 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-5ccd7b-655967d999-p8xqs"] Nov 01 16:33:29.018591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.018570 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4"] Nov 01 16:33:29.021407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.021388 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-5ccd7b-6f9b864d6c-2dft4"] Nov 01 16:33:29.257001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.256951 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:29.451129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.451092 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9"] Nov 01 16:33:29.464941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.464910 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6"] Nov 01 16:33:29.521508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.521406 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:29.620599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.620538 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:29.968951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:29.968900 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:30.026153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026120 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc"] Nov 01 16:33:30.026677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026664 4573 cpu_manager.go:414] "containerMap: removing container" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" Nov 01 16:33:30.026719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026679 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" Nov 01 16:33:30.026719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026711 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f82a8513-36f7-4799-a7c7-0c685a52631d" containerName="payload-processing" Nov 01 16:33:30.026719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026716 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f82a8513-36f7-4799-a7c7-0c685a52631d" containerName="payload-processing" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026724 4573 cpu_manager.go:414] "containerMap: removing container" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026729 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026738 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026742 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026767 4573 cpu_manager.go:414] "containerMap: removing container" podUID="fde1a22e-1a29-40fe-ae79-9b3f717fc392" containerName="payload-pre-processing" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026772 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fde1a22e-1a29-40fe-ae79-9b3f717fc392" containerName="payload-pre-processing" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026780 4573 cpu_manager.go:414] "containerMap: removing container" podUID="cdfe623c-be76-460e-a5f4-8d61b94e3fd8" containerName="revoke-keys" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026784 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdfe623c-be76-460e-a5f4-8d61b94e3fd8" containerName="revoke-keys" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026790 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3698bc5b-a020-4f29-8164-00d35322a558" containerName="revoke-keys" Nov 01 16:33:30.026868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026794 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3698bc5b-a020-4f29-8164-00d35322a558" containerName="revoke-keys" Nov 01 16:33:30.027163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026882 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f82a8513-36f7-4799-a7c7-0c685a52631d" containerName="payload-processing" Nov 01 16:33:30.027163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026891 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e6721a2-bcda-4eab-a726-90812e0dfb06" containerName="istio-proxy" Nov 01 16:33:30.027163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026899 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3ec6a5-f442-4335-b93c-b6bcea7de99f" containerName="istio-proxy" Nov 01 16:33:30.027163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026906 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="fde1a22e-1a29-40fe-ae79-9b3f717fc392" containerName="payload-pre-processing" Nov 01 16:33:30.027163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026912 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdfe623c-be76-460e-a5f4-8d61b94e3fd8" containerName="revoke-keys" Nov 01 16:33:30.027163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026918 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3698bc5b-a020-4f29-8164-00d35322a558" containerName="revoke-keys" Nov 01 16:33:30.027163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.026926 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="12add1b5-31ff-4c5c-b33b-cf8ffd361bf0" containerName="istio-proxy" Nov 01 16:33:30.031900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.031886 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:30.038690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.038607 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f4a265\"" Nov 01 16:33:30.038690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.038657 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f4a265-dockercfg-t4tm8\"" Nov 01 16:33:30.039250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.039223 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc"] Nov 01 16:33:30.054936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.054908 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:30.065563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.065533 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-624zx\" (UniqueName: \"kubernetes.io/projected/96a8bdf5-8654-4e4f-a2c7-3974af866440-kube-api-access-624zx\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc\" (UID: \"96a8bdf5-8654-4e4f-a2c7-3974af866440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:30.065687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.065598 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a8bdf5-8654-4e4f-a2c7-3974af866440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc\" (UID: \"96a8bdf5-8654-4e4f-a2c7-3974af866440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:30.180772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.171549 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-624zx\" (UniqueName: \"kubernetes.io/projected/96a8bdf5-8654-4e4f-a2c7-3974af866440-kube-api-access-624zx\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc\" (UID: \"96a8bdf5-8654-4e4f-a2c7-3974af866440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:30.180772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.171636 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a8bdf5-8654-4e4f-a2c7-3974af866440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc\" (UID: \"96a8bdf5-8654-4e4f-a2c7-3974af866440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:30.180772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.172841 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a8bdf5-8654-4e4f-a2c7-3974af866440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc\" (UID: \"96a8bdf5-8654-4e4f-a2c7-3974af866440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:30.198444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.197079 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-624zx\" (UniqueName: \"kubernetes.io/projected/96a8bdf5-8654-4e4f-a2c7-3974af866440-kube-api-access-624zx\") pod \"payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc\" (UID: \"96a8bdf5-8654-4e4f-a2c7-3974af866440\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:30.279233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.279150 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:30.348800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.348711 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:30.460276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.460247 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f82a8513-36f7-4799-a7c7-0c685a52631d" path="/var/lib/kubelet/pods/f82a8513-36f7-4799-a7c7-0c685a52631d/volumes" Nov 01 16:33:30.460626 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.460612 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fde1a22e-1a29-40fe-ae79-9b3f717fc392" path="/var/lib/kubelet/pods/fde1a22e-1a29-40fe-ae79-9b3f717fc392/volumes" Nov 01 16:33:30.634449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.634420 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc"] Nov 01 16:33:30.634590 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:30.634569 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a8bdf5_8654_4e4f_a2c7_3974af866440.slice/crio-fb60830b5285396972c3bc879ca0850df1d63e11487ccf335130ece32fc24fd2 WatchSource:0}: Error finding container fb60830b5285396972c3bc879ca0850df1d63e11487ccf335130ece32fc24fd2: Status 404 returned error can't find the container with id fb60830b5285396972c3bc879ca0850df1d63e11487ccf335130ece32fc24fd2 Nov 01 16:33:30.832460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.832421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:31.003825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:30.978879 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:31.042339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.041274 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" event={"ID":"96a8bdf5-8654-4e4f-a2c7-3974af866440","Type":"ContainerStarted","Data":"f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee"} Nov 01 16:33:31.042339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.041344 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" event={"ID":"96a8bdf5-8654-4e4f-a2c7-3974af866440","Type":"ContainerStarted","Data":"fb60830b5285396972c3bc879ca0850df1d63e11487ccf335130ece32fc24fd2"} Nov 01 16:33:31.042339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.042265 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:31.062699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.062651 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" podStartSLOduration=1.062636415 podStartE2EDuration="1.062636415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:31.058556421 +0000 UTC m=+2748.761722358" watchObservedRunningTime="2026-11-01 16:33:31.062636415 +0000 UTC m=+2748.765802351" Nov 01 16:33:31.065071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.065044 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:31.131877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.131842 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf"] Nov 01 16:33:31.134999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.134976 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-bd831a-grrbf"] Nov 01 16:33:31.228647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.228616 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:31.572968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:31.572934 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:32.047325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:32.047277 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:32.457919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:32.457885 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a431f4fc-4c91-49b0-8ee4-a7bff87f8580" path="/var/lib/kubelet/pods/a431f4fc-4c91-49b0-8ee4-a7bff87f8580/volumes" Nov 01 16:33:32.461802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:32.461779 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:32.738905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:32.738836 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:33.005105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:33.005028 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z"] Nov 01 16:33:33.201427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:33.201383 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:33.287296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:33.287215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:33.344482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:33.344433 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" podUID="c5b94c21-751f-4484-9cfd-8100407cb40f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:33.774535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:33.774501 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:34.319682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.319650 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd"] Nov 01 16:33:34.320047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.319876 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" podUID="853b6f9e-33a9-448c-b988-155a08666a2f" containerName="maas-api" containerID="cri-o://3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f" gracePeriod=30 Nov 01 16:33:34.356937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.356908 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:34.451600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.451552 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" podUID="75996380-b13c-4d2a-8a70-494c0bf2ac06" containerName="payload-processing" containerID="cri-o://0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170" gracePeriod=25 Nov 01 16:33:34.467508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.466963 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" podUID="5eba7eb6-54bb-45e1-9522-886df60e939f" containerName="payload-pre-processing" containerID="cri-o://0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373" gracePeriod=25 Nov 01 16:33:34.558985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.537551 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:34.622166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.622126 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:34.637331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.637298 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4"] Nov 01 16:33:34.640745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.640718 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-c8537c-lcmd4"] Nov 01 16:33:34.670709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.670689 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:33:34.722514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.722481 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:34.739875 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.739846 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-metrics-certs\") pod \"853b6f9e-33a9-448c-b988-155a08666a2f\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " Nov 01 16:33:34.740047 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.739963 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/853b6f9e-33a9-448c-b988-155a08666a2f-kube-api-access-dv6dz\" (UniqueName: \"kubernetes.io/projected/853b6f9e-33a9-448c-b988-155a08666a2f-kube-api-access-dv6dz\") pod \"853b6f9e-33a9-448c-b988-155a08666a2f\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " Nov 01 16:33:34.740119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.740080 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-maas-api-tls\") pod \"853b6f9e-33a9-448c-b988-155a08666a2f\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " Nov 01 16:33:34.740188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.740133 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/853b6f9e-33a9-448c-b988-155a08666a2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853b6f9e-33a9-448c-b988-155a08666a2f-openshift-service-ca\") pod \"853b6f9e-33a9-448c-b988-155a08666a2f\" (UID: \"853b6f9e-33a9-448c-b988-155a08666a2f\") " Nov 01 16:33:34.740557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.740529 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853b6f9e-33a9-448c-b988-155a08666a2f-openshift-service-ca" pod "853b6f9e-33a9-448c-b988-155a08666a2f" (UID: "853b6f9e-33a9-448c-b988-155a08666a2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:34.742419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.742393 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-metrics-certs" pod "853b6f9e-33a9-448c-b988-155a08666a2f" (UID: "853b6f9e-33a9-448c-b988-155a08666a2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:34.742522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.742468 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-maas-api-tls" pod "853b6f9e-33a9-448c-b988-155a08666a2f" (UID: "853b6f9e-33a9-448c-b988-155a08666a2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:34.742919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.742893 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853b6f9e-33a9-448c-b988-155a08666a2f-kube-api-access-dv6dz" pod "853b6f9e-33a9-448c-b988-155a08666a2f" (UID: "853b6f9e-33a9-448c-b988-155a08666a2f"). InnerVolumeSpecName "kube-api-access-dv6dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:34.841682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.841642 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:34.841682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.841674 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/853b6f9e-33a9-448c-b988-155a08666a2f-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:34.841682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.841685 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/853b6f9e-33a9-448c-b988-155a08666a2f-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:34.842174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:34.841695 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6dz\" (UniqueName: \"kubernetes.io/projected/853b6f9e-33a9-448c-b988-155a08666a2f-kube-api-access-dv6dz\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:35.075820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.075717 4573 generic.go:356] "Generic (PLEG): container finished" podID="853b6f9e-33a9-448c-b988-155a08666a2f" containerID="3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f" exitCode=0 Nov 01 16:33:35.075820 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.075801 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" Nov 01 16:33:35.076064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.075822 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" event={"ID":"853b6f9e-33a9-448c-b988-155a08666a2f","Type":"ContainerDied","Data":"3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f"} Nov 01 16:33:35.076064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.075894 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd" event={"ID":"853b6f9e-33a9-448c-b988-155a08666a2f","Type":"ContainerDied","Data":"aa638a854deda35be0d25d14970fe40587a71f00111ff5807bd68126e04fe860"} Nov 01 16:33:35.076064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.075923 4573 scope.go:122] "RemoveContainer" containerID="3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f" Nov 01 16:33:35.088492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.088474 4573 scope.go:122] "RemoveContainer" containerID="3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f" Nov 01 16:33:35.088846 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:35.088815 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f\": container with ID starting with 3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f not found: ID does not exist" containerID="3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f" Nov 01 16:33:35.088954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.088856 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f"} err="failed to get container status \"3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f\": rpc error: code = NotFound desc = could not find container \"3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f\": container with ID starting with 3ac0ad7b6a35c3349ffccb8c55ed99e147f22622f4efa428251e8a1e3589bf0f not found: ID does not exist" Nov 01 16:33:35.100152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.100128 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd"] Nov 01 16:33:35.103317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.103293 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-c8537c-5fb97495fc-fp8bd"] Nov 01 16:33:35.111306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.111287 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:35.455770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.455000 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8"] Nov 01 16:33:35.455770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.455695 4573 cpu_manager.go:414] "containerMap: removing container" podUID="853b6f9e-33a9-448c-b988-155a08666a2f" containerName="maas-api" Nov 01 16:33:35.455770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.455716 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="853b6f9e-33a9-448c-b988-155a08666a2f" containerName="maas-api" Nov 01 16:33:35.457167 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.457141 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="853b6f9e-33a9-448c-b988-155a08666a2f" containerName="maas-api" Nov 01 16:33:35.462113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.462073 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.474022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.473997 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 16:33:35.474143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.474081 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f4a265-serving-cert\"" Nov 01 16:33:35.474143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.474140 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 16:33:35.478609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.478581 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8"] Nov 01 16:33:35.507706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.507669 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:35.550611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.550584 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mvkn\" (UniqueName: \"kubernetes.io/projected/2ff6f72e-575a-4019-9d99-2232e3638fdc-kube-api-access-8mvkn\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.550799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.550783 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-metrics-certs\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.550869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.550825 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6f72e-575a-4019-9d99-2232e3638fdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.550983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.550956 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.621449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.619005 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:35.651759 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.651713 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-metrics-certs\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.651885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.651823 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6f72e-575a-4019-9d99-2232e3638fdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.651885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.651872 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.651980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.651922 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mvkn\" (UniqueName: \"kubernetes.io/projected/2ff6f72e-575a-4019-9d99-2232e3638fdc-kube-api-access-8mvkn\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.652558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.652533 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6f72e-575a-4019-9d99-2232e3638fdc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.654272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.654248 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.654355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.654342 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-metrics-certs\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.655569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.655549 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 16:33:35.659714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.659692 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mvkn\" (UniqueName: \"kubernetes.io/projected/2ff6f72e-575a-4019-9d99-2232e3638fdc-kube-api-access-8mvkn\") pod \"maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.753079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.752998 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b94c21-751f-4484-9cfd-8100407cb40f-kube-api-access-mccmv\" (UniqueName: \"kubernetes.io/projected/c5b94c21-751f-4484-9cfd-8100407cb40f-kube-api-access-mccmv\") pod \"c5b94c21-751f-4484-9cfd-8100407cb40f\" (UID: \"c5b94c21-751f-4484-9cfd-8100407cb40f\") " Nov 01 16:33:35.753243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.753161 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b94c21-751f-4484-9cfd-8100407cb40f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b94c21-751f-4484-9cfd-8100407cb40f-plugins-config-volume\") pod \"c5b94c21-751f-4484-9cfd-8100407cb40f\" (UID: \"c5b94c21-751f-4484-9cfd-8100407cb40f\") " Nov 01 16:33:35.753554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.753530 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b94c21-751f-4484-9cfd-8100407cb40f-plugins-config-volume" pod "c5b94c21-751f-4484-9cfd-8100407cb40f" (UID: "c5b94c21-751f-4484-9cfd-8100407cb40f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:35.755026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.755007 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b94c21-751f-4484-9cfd-8100407cb40f-kube-api-access-mccmv" pod "c5b94c21-751f-4484-9cfd-8100407cb40f" (UID: "c5b94c21-751f-4484-9cfd-8100407cb40f"). InnerVolumeSpecName "kube-api-access-mccmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:35.831544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.831516 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:35.854478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.854448 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mccmv\" (UniqueName: \"kubernetes.io/projected/c5b94c21-751f-4484-9cfd-8100407cb40f-kube-api-access-mccmv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:35.854478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:35.854475 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b94c21-751f-4484-9cfd-8100407cb40f-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:36.088240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.088128 4573 generic.go:356] "Generic (PLEG): container finished" podID="c5b94c21-751f-4484-9cfd-8100407cb40f" containerID="c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa" exitCode=0 Nov 01 16:33:36.088459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.088236 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" event={"ID":"c5b94c21-751f-4484-9cfd-8100407cb40f","Type":"ContainerDied","Data":"c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa"} Nov 01 16:33:36.088459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.088328 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" event={"ID":"c5b94c21-751f-4484-9cfd-8100407cb40f","Type":"ContainerDied","Data":"b48e7e2abf3ee4dbcea203a387db1000f5cbee1cd5c2cb3c70dc696147602005"} Nov 01 16:33:36.088459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.088363 4573 scope.go:122] "RemoveContainer" containerID="c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa" Nov 01 16:33:36.088651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.088555 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9" Nov 01 16:33:36.108586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.108562 4573 scope.go:122] "RemoveContainer" containerID="c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa" Nov 01 16:33:36.108952 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:36.108930 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa\": container with ID starting with c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa not found: ID does not exist" containerID="c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa" Nov 01 16:33:36.109026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.108963 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa"} err="failed to get container status \"c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa\": rpc error: code = NotFound desc = could not find container \"c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa\": container with ID starting with c8ecec1cfded43dfdfaf90f37d3c4ddfe6416c15da04964f6b632f45749047aa not found: ID does not exist" Nov 01 16:33:36.119141 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.119117 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9"] Nov 01 16:33:36.126192 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:36.126164 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff6f72e_575a_4019_9d99_2232e3638fdc.slice/crio-7375a932e2ec5e8c8ea56263a5ec9c2244826bc5c8100e73e962300bdefb2a95 WatchSource:0}: Error finding container 7375a932e2ec5e8c8ea56263a5ec9c2244826bc5c8100e73e962300bdefb2a95: Status 404 returned error can't find the container with id 7375a932e2ec5e8c8ea56263a5ec9c2244826bc5c8100e73e962300bdefb2a95 Nov 01 16:33:36.126595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.126565 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8"] Nov 01 16:33:36.131329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.131263 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bd831a-544dc6b955-mn6k9"] Nov 01 16:33:36.320511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.320475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:36.457969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.457933 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="853b6f9e-33a9-448c-b988-155a08666a2f" path="/var/lib/kubelet/pods/853b6f9e-33a9-448c-b988-155a08666a2f/volumes" Nov 01 16:33:36.458389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.458374 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b94c21-751f-4484-9cfd-8100407cb40f" path="/var/lib/kubelet/pods/c5b94c21-751f-4484-9cfd-8100407cb40f/volumes" Nov 01 16:33:36.458836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.458821 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdfe623c-be76-460e-a5f4-8d61b94e3fd8" path="/var/lib/kubelet/pods/cdfe623c-be76-460e-a5f4-8d61b94e3fd8/volumes" Nov 01 16:33:36.660337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.660305 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" podUID="5eba7eb6-54bb-45e1-9522-886df60e939f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:36.981088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:36.981053 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:37.097928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:37.097884 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" event={"ID":"2ff6f72e-575a-4019-9d99-2232e3638fdc","Type":"ContainerStarted","Data":"e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762"} Nov 01 16:33:37.098105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:37.097950 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" event={"ID":"2ff6f72e-575a-4019-9d99-2232e3638fdc","Type":"ContainerStarted","Data":"7375a932e2ec5e8c8ea56263a5ec9c2244826bc5c8100e73e962300bdefb2a95"} Nov 01 16:33:37.098105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:37.097986 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:37.118778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:37.118718 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" podStartSLOduration=1.689105866 podStartE2EDuration="2.118705271s" totalImagesPullingTime="429.599405ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:37.116579161 +0000 UTC m=+2754.819745097" watchObservedRunningTime="2026-11-01 16:33:37.118705271 +0000 UTC m=+2754.821871208" Nov 01 16:33:37.667794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:37.667739 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" podUID="75996380-b13c-4d2a-8a70-494c0bf2ac06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:37.670290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:37.670264 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:37.990017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:37.989917 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:38.006310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:38.006276 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" podUID="e416ad18-b9a1-4e75-8d3a-db6cf4a20203" containerName="payload-pre-processing" containerID="cri-o://a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157" gracePeriod=25 Nov 01 16:33:38.095719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:38.095676 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:38.297395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:38.297287 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs"] Nov 01 16:33:38.297622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:38.297526 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" containerID="cri-o://b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9" gracePeriod=30 Nov 01 16:33:38.317968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:38.317933 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:38.417300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:38.417265 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:38.969835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:38.969790 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:33:39.028120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:39.028088 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:39.206206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:39.206167 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h"] Nov 01 16:33:39.206709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:39.206676 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:39.215573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:39.215540 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f01de5-jbq6h"] Nov 01 16:33:39.290831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:39.285871 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:39.419847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:39.419815 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:39.664405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:39.664372 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:40.376310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:40.376278 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:40.474832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:40.474784 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3698bc5b-a020-4f29-8164-00d35322a558" path="/var/lib/kubelet/pods/3698bc5b-a020-4f29-8164-00d35322a558/volumes" Nov 01 16:33:40.756280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:40.756189 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" podUID="e416ad18-b9a1-4e75-8d3a-db6cf4a20203" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:41.167105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:41.167068 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:41.447386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:41.447307 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:41.587243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:41.587209 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:41.682344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:41.682308 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:41.780235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:41.780153 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:41.919501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:41.919465 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:42.020632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.020601 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:42.119854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.119824 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:42.236362 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.236326 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:42.685024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.684987 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:42.909953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.909925 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp"] Nov 01 16:33:42.910488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.910475 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c5b94c21-751f-4484-9cfd-8100407cb40f" containerName="payload-processing" Nov 01 16:33:42.910546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.910490 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b94c21-751f-4484-9cfd-8100407cb40f" containerName="payload-processing" Nov 01 16:33:42.910585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.910574 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b94c21-751f-4484-9cfd-8100407cb40f" containerName="payload-processing" Nov 01 16:33:42.914008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.913991 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:42.921422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.921391 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp"] Nov 01 16:33:42.958042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.957974 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:42.958042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.958008 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:42.958266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.958173 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sf9b\" (UniqueName: \"kubernetes.io/projected/4c4acb5f-73ff-4468-926d-29ab31b2857f-kube-api-access-5sf9b\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:42.990601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.990573 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:33:42.990805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:42.990790 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:33:43.035185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.035158 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:43.059519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.059490 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sf9b\" (UniqueName: \"kubernetes.io/projected/4c4acb5f-73ff-4468-926d-29ab31b2857f-kube-api-access-5sf9b\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:43.059673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.059573 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:43.059673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.059594 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:43.060356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.060333 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:43.062003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.061974 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:43.063216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.063196 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:43.067847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.067826 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sf9b\" (UniqueName: \"kubernetes.io/projected/4c4acb5f-73ff-4468-926d-29ab31b2857f-kube-api-access-5sf9b\") pod \"maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:43.111842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.111810 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:33:43.230618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.230537 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:43.558720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.558695 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:33:43.574231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.574201 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp"] Nov 01 16:33:43.574913 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:43.574886 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4acb5f_73ff_4468_926d_29ab31b2857f.slice/crio-a141ee35af3665e25e406cfb35296213cb829243d7ac0f0cc8b9f13dbffb5e86 WatchSource:0}: Error finding container a141ee35af3665e25e406cfb35296213cb829243d7ac0f0cc8b9f13dbffb5e86: Status 404 returned error can't find the container with id a141ee35af3665e25e406cfb35296213cb829243d7ac0f0cc8b9f13dbffb5e86 Nov 01 16:33:43.669109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669074 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istiod-ca-cert\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669114 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-podinfo\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669143 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-data\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669209 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-kube-api-access-t2kc5\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-kube-api-access-t2kc5\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669285 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-certs\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669318 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-token\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-token\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669400 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-envoy\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669503 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-socket\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669529 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-credential-socket\") pod \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\" (UID: \"f4bf3f4c-76f8-4434-8015-b6515fd018d3\") " Nov 01 16:33:43.669607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669558 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-data" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.669607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669577 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-certs" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.669607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669587 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istiod-ca-cert" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:43.670071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669835 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-credential-socket" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.670071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.669849 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-socket" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.670204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.670135 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.670204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.670154 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.670204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.670168 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.670204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.670182 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.670204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.670196 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.671469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.671445 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-podinfo" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:43.671571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.671508 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-kube-api-access-t2kc5" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "kube-api-access-t2kc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:43.671983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.671965 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-envoy" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:43.672038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.671970 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-token" pod "f4bf3f4c-76f8-4434-8015-b6515fd018d3" (UID: "f4bf3f4c-76f8-4434-8015-b6515fd018d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:43.771888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.771857 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.771888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.771886 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2kc5\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-kube-api-access-t2kc5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.772277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.771897 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.772277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.771906 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4bf3f4c-76f8-4434-8015-b6515fd018d3-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:43.854569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:43.854467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:44.036367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.036333 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:44.164914 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.164879 4573 generic.go:356] "Generic (PLEG): container finished" podID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerID="b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9" exitCode=0 Nov 01 16:33:44.165100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.164948 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" Nov 01 16:33:44.165100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.164962 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" event={"ID":"f4bf3f4c-76f8-4434-8015-b6515fd018d3","Type":"ContainerDied","Data":"b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9"} Nov 01 16:33:44.165100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.165069 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs" event={"ID":"f4bf3f4c-76f8-4434-8015-b6515fd018d3","Type":"ContainerDied","Data":"504d5716b36c5cecb92822154b2e508480120a49351bfb14c9c2d977f37076f4"} Nov 01 16:33:44.165230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.165107 4573 scope.go:122] "RemoveContainer" containerID="b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9" Nov 01 16:33:44.166897 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.166867 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" event={"ID":"4c4acb5f-73ff-4468-926d-29ab31b2857f","Type":"ContainerStarted","Data":"f9286f5431abdabd1625990a3139f8a096ad2252862924c1fcb1e04731356d05"} Nov 01 16:33:44.167039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.166917 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" event={"ID":"4c4acb5f-73ff-4468-926d-29ab31b2857f","Type":"ContainerStarted","Data":"a141ee35af3665e25e406cfb35296213cb829243d7ac0f0cc8b9f13dbffb5e86"} Nov 01 16:33:44.176911 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.176896 4573 scope.go:122] "RemoveContainer" containerID="b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9" Nov 01 16:33:44.177179 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:44.177162 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9\": container with ID starting with b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9 not found: ID does not exist" containerID="b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9" Nov 01 16:33:44.177229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.177188 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9"} err="failed to get container status \"b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9\": rpc error: code = NotFound desc = could not find container \"b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9\": container with ID starting with b0420b326876112ede63453bffc1f87ad6052af434647a80051406630772bfd9 not found: ID does not exist" Nov 01 16:33:44.186798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.186739 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" podStartSLOduration=2.186728477 podStartE2EDuration="2.186728477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:44.183147426 +0000 UTC m=+2761.886313365" watchObservedRunningTime="2026-11-01 16:33:44.186728477 +0000 UTC m=+2761.889894413" Nov 01 16:33:44.197082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.197062 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs"] Nov 01 16:33:44.201463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.201439 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bd831a-openshift-default-5b796d5fbf-5wdfs"] Nov 01 16:33:44.457843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.457768 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" path="/var/lib/kubelet/pods/f4bf3f4c-76f8-4434-8015-b6515fd018d3/volumes" Nov 01 16:33:44.773779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:44.773670 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:45.176144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:45.176112 4573 generic.go:356] "Generic (PLEG): container finished" podID="4c4acb5f-73ff-4468-926d-29ab31b2857f" containerID="f9286f5431abdabd1625990a3139f8a096ad2252862924c1fcb1e04731356d05" exitCode=0 Nov 01 16:33:45.176321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:45.176197 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" event={"ID":"4c4acb5f-73ff-4468-926d-29ab31b2857f","Type":"ContainerDied","Data":"f9286f5431abdabd1625990a3139f8a096ad2252862924c1fcb1e04731356d05"} Nov 01 16:33:45.672032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:45.671968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:45.895414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:45.895384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:45.992418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:45.992335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:46.205934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.205900 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6"] Nov 01 16:33:46.206163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.206133 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" containerID="cri-o://5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08" gracePeriod=30 Nov 01 16:33:46.239444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.239387 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:46.374480 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.374457 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:46.506317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.506228 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-xks-ca\") pod \"4c4acb5f-73ff-4468-926d-29ab31b2857f\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " Nov 01 16:33:46.506477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.506326 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c4acb5f-73ff-4468-926d-29ab31b2857f-kube-api-access-5sf9b\" (UniqueName: \"kubernetes.io/projected/4c4acb5f-73ff-4468-926d-29ab31b2857f-kube-api-access-5sf9b\") pod \"4c4acb5f-73ff-4468-926d-29ab31b2857f\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " Nov 01 16:33:46.506477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.506385 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-openshift-ca\") pod \"4c4acb5f-73ff-4468-926d-29ab31b2857f\" (UID: \"4c4acb5f-73ff-4468-926d-29ab31b2857f\") " Nov 01 16:33:46.506810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.506745 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-openshift-ca" pod "4c4acb5f-73ff-4468-926d-29ab31b2857f" (UID: "4c4acb5f-73ff-4468-926d-29ab31b2857f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:46.508268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.508247 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-xks-ca" pod "4c4acb5f-73ff-4468-926d-29ab31b2857f" (UID: "4c4acb5f-73ff-4468-926d-29ab31b2857f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:46.508403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.508385 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c4acb5f-73ff-4468-926d-29ab31b2857f-kube-api-access-5sf9b" pod "4c4acb5f-73ff-4468-926d-29ab31b2857f" (UID: "4c4acb5f-73ff-4468-926d-29ab31b2857f"). InnerVolumeSpecName "kube-api-access-5sf9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:46.607818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.607740 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:46.607818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.607821 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c4acb5f-73ff-4468-926d-29ab31b2857f-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:46.608113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.607837 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sf9b\" (UniqueName: \"kubernetes.io/projected/4c4acb5f-73ff-4468-926d-29ab31b2857f-kube-api-access-5sf9b\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:46.659118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.659073 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" podUID="5eba7eb6-54bb-45e1-9522-886df60e939f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:46.708236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.699810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:46.790689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.790609 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7"] Nov 01 16:33:46.791182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.791168 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" Nov 01 16:33:46.791242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.791183 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" Nov 01 16:33:46.791242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.791198 4573 cpu_manager.go:414] "containerMap: removing container" podUID="4c4acb5f-73ff-4468-926d-29ab31b2857f" containerName="revoke-keys" Nov 01 16:33:46.791242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.791203 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c4acb5f-73ff-4468-926d-29ab31b2857f" containerName="revoke-keys" Nov 01 16:33:46.791342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.791275 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4bf3f4c-76f8-4434-8015-b6515fd018d3" containerName="istio-proxy" Nov 01 16:33:46.791342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.791286 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c4acb5f-73ff-4468-926d-29ab31b2857f" containerName="revoke-keys" Nov 01 16:33:46.794703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.794683 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:46.797840 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.797817 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:46.804572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.804549 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7"] Nov 01 16:33:46.911196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.911157 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kspm4\" (UniqueName: \"kubernetes.io/projected/245b7454-2574-408f-bf01-8dd608326607-kube-api-access-kspm4\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:46.911614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.911285 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245b7454-2574-408f-bf01-8dd608326607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:46.911614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:46.911380 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245b7454-2574-408f-bf01-8dd608326607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:47.012207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.012174 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245b7454-2574-408f-bf01-8dd608326607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:47.012421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.012248 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245b7454-2574-408f-bf01-8dd608326607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:47.012421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.012371 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kspm4\" (UniqueName: \"kubernetes.io/projected/245b7454-2574-408f-bf01-8dd608326607-kube-api-access-kspm4\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:47.012931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.012906 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245b7454-2574-408f-bf01-8dd608326607-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:47.014576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.014554 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245b7454-2574-408f-bf01-8dd608326607-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:47.019894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.019872 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kspm4\" (UniqueName: \"kubernetes.io/projected/245b7454-2574-408f-bf01-8dd608326607-kube-api-access-kspm4\") pod \"maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:47.113163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.113131 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:47.195924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.195902 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" Nov 01 16:33:47.196064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.195891 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp" event={"ID":"4c4acb5f-73ff-4468-926d-29ab31b2857f","Type":"ContainerDied","Data":"a141ee35af3665e25e406cfb35296213cb829243d7ac0f0cc8b9f13dbffb5e86"} Nov 01 16:33:47.196064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.195987 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a141ee35af3665e25e406cfb35296213cb829243d7ac0f0cc8b9f13dbffb5e86" Nov 01 16:33:47.401336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.401306 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7"] Nov 01 16:33:47.402268 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:47.402241 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245b7454_2574_408f_bf01_8dd608326607.slice/crio-08200e64de122c5b82a0fa7dd715bc77d3d4b48fd403d404c1977c57a8a62fea WatchSource:0}: Error finding container 08200e64de122c5b82a0fa7dd715bc77d3d4b48fd403d404c1977c57a8a62fea: Status 404 returned error can't find the container with id 08200e64de122c5b82a0fa7dd715bc77d3d4b48fd403d404c1977c57a8a62fea Nov 01 16:33:47.659101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.659025 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56"] Nov 01 16:33:47.667382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.667344 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" podUID="75996380-b13c-4d2a-8a70-494c0bf2ac06" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:47.717525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.717497 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:47.807085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:47.807049 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:48.205121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:48.205069 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" event={"ID":"245b7454-2574-408f-bf01-8dd608326607","Type":"ContainerStarted","Data":"1fdb2220044375484ae10740e434873c983c3e98a28ae82bc4e288f4fcc74533"} Nov 01 16:33:48.205577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:48.205152 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" event={"ID":"245b7454-2574-408f-bf01-8dd608326607","Type":"ContainerStarted","Data":"08200e64de122c5b82a0fa7dd715bc77d3d4b48fd403d404c1977c57a8a62fea"} Nov 01 16:33:48.221902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:48.221860 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" podStartSLOduration=2.2218471810000002 podStartE2EDuration="2.221847181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:33:48.220031522 +0000 UTC m=+2765.923197459" watchObservedRunningTime="2026-11-01 16:33:48.221847181 +0000 UTC m=+2765.925013155" Nov 01 16:33:48.446777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:48.446712 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:49.185449 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.185413 4573 patch_prober.go:29] interesting pod/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:33:49.185622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.185467 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:33:49.205280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.205058 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:49.218189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.218149 4573 generic.go:356] "Generic (PLEG): container finished" podID="245b7454-2574-408f-bf01-8dd608326607" containerID="1fdb2220044375484ae10740e434873c983c3e98a28ae82bc4e288f4fcc74533" exitCode=0 Nov 01 16:33:49.218334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.218210 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" event={"ID":"245b7454-2574-408f-bf01-8dd608326607","Type":"ContainerDied","Data":"1fdb2220044375484ae10740e434873c983c3e98a28ae82bc4e288f4fcc74533"} Nov 01 16:33:49.495133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.495056 4573 scope.go:122] "RemoveContainer" containerID="a2cc94efc3fdf0a83a25249f66e1b80c580c51bbb61e25620d53c9d0f82f0e73" Nov 01 16:33:49.505369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.505347 4573 scope.go:122] "RemoveContainer" containerID="e48378aab9942188905c4ca7b84132f6358b3f133514329d0fc645a536ee35ac" Nov 01 16:33:49.515098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.515078 4573 scope.go:122] "RemoveContainer" containerID="82343cfa3ef03cee231710c540e1bcd5063de9d7b93899129878974a79da5c77" Nov 01 16:33:49.623582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:49.623557 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:50.041461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.041426 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:50.365921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.365897 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:50.451990 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.451955 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245b7454-2574-408f-bf01-8dd608326607-kube-api-access-kspm4\" (UniqueName: \"kubernetes.io/projected/245b7454-2574-408f-bf01-8dd608326607-kube-api-access-kspm4\") pod \"245b7454-2574-408f-bf01-8dd608326607\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " Nov 01 16:33:50.452176 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.452078 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245b7454-2574-408f-bf01-8dd608326607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245b7454-2574-408f-bf01-8dd608326607-maas-api-xks-ca\") pod \"245b7454-2574-408f-bf01-8dd608326607\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " Nov 01 16:33:50.452266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.452244 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245b7454-2574-408f-bf01-8dd608326607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245b7454-2574-408f-bf01-8dd608326607-maas-api-openshift-ca\") pod \"245b7454-2574-408f-bf01-8dd608326607\" (UID: \"245b7454-2574-408f-bf01-8dd608326607\") " Nov 01 16:33:50.452796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.452703 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245b7454-2574-408f-bf01-8dd608326607-maas-api-openshift-ca" pod "245b7454-2574-408f-bf01-8dd608326607" (UID: "245b7454-2574-408f-bf01-8dd608326607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:50.454281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.454255 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245b7454-2574-408f-bf01-8dd608326607-maas-api-xks-ca" pod "245b7454-2574-408f-bf01-8dd608326607" (UID: "245b7454-2574-408f-bf01-8dd608326607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:33:50.454401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.454325 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245b7454-2574-408f-bf01-8dd608326607-kube-api-access-kspm4" pod "245b7454-2574-408f-bf01-8dd608326607" (UID: "245b7454-2574-408f-bf01-8dd608326607"). InnerVolumeSpecName "kube-api-access-kspm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:50.554240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.554210 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/245b7454-2574-408f-bf01-8dd608326607-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:50.554370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.554243 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kspm4\" (UniqueName: \"kubernetes.io/projected/245b7454-2574-408f-bf01-8dd608326607-kube-api-access-kspm4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:50.554370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.554259 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/245b7454-2574-408f-bf01-8dd608326607-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:50.744092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.744059 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:50.756339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:50.756311 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" podUID="e416ad18-b9a1-4e75-8d3a-db6cf4a20203" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:33:51.234061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.234042 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" Nov 01 16:33:51.234161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.234080 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7" event={"ID":"245b7454-2574-408f-bf01-8dd608326607","Type":"ContainerDied","Data":"08200e64de122c5b82a0fa7dd715bc77d3d4b48fd403d404c1977c57a8a62fea"} Nov 01 16:33:51.234161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.234132 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08200e64de122c5b82a0fa7dd715bc77d3d4b48fd403d404c1977c57a8a62fea" Nov 01 16:33:51.439502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.439466 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:51.471340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.471286 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:33:51.565660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.565614 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-socket\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.565836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.565677 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-certs\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.565836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.565735 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-istio-token\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-istio-token\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.565836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.565803 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-kube-api-access-gdxwm\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-kube-api-access-gdxwm\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.566027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.565962 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-socket" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.566027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.565975 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-data\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.566140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566053 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f540387f-07a4-4601-b63a-99922263287f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540387f-07a4-4601-b63a-99922263287f-istio-podinfo\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.566140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566056 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-certs" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.566252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566210 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-credential-socket\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.566307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566248 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-envoy\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.566307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566249 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-data" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.566307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566287 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f540387f-07a4-4601-b63a-99922263287f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540387f-07a4-4601-b63a-99922263287f-istiod-ca-cert\") pod \"f540387f-07a4-4601-b63a-99922263287f\" (UID: \"f540387f-07a4-4601-b63a-99922263287f\") " Nov 01 16:33:51.566468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566449 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-credential-socket" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.566682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566654 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f540387f-07a4-4601-b63a-99922263287f-istiod-ca-cert" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:51.566919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566897 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.567004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566923 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.567004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566941 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.567004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566956 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f540387f-07a4-4601-b63a-99922263287f-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.567004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.566971 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.568561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.568535 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-kube-api-access-gdxwm" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "kube-api-access-gdxwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:51.568561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.568555 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-istio-token" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:51.568964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.568945 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-envoy" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:33:51.569130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.569113 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f540387f-07a4-4601-b63a-99922263287f-istio-podinfo" pod "f540387f-07a4-4601-b63a-99922263287f" (UID: "f540387f-07a4-4601-b63a-99922263287f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:33:51.618894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.618865 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg"] Nov 01 16:33:51.667622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.667588 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.667622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.667621 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdxwm\" (UniqueName: \"kubernetes.io/projected/f540387f-07a4-4601-b63a-99922263287f-kube-api-access-gdxwm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.667854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.667633 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f540387f-07a4-4601-b63a-99922263287f-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.667854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.667642 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f540387f-07a4-4601-b63a-99922263287f-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:51.711215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:51.711170 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:52.167631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.167591 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:52.245715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.245676 4573 generic.go:356] "Generic (PLEG): container finished" podID="f540387f-07a4-4601-b63a-99922263287f" containerID="5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08" exitCode=0 Nov 01 16:33:52.245994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.245973 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" event={"ID":"f540387f-07a4-4601-b63a-99922263287f","Type":"ContainerDied","Data":"5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08"} Nov 01 16:33:52.246128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.246065 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" event={"ID":"f540387f-07a4-4601-b63a-99922263287f","Type":"ContainerDied","Data":"07f97cd8f4d9cfe27c64848a9efe530bd3a87723917f3f75fd2cf1ade66f37ab"} Nov 01 16:33:52.246128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.246095 4573 scope.go:122] "RemoveContainer" containerID="5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08" Nov 01 16:33:52.246297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.246283 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6" Nov 01 16:33:52.263388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.263368 4573 scope.go:122] "RemoveContainer" containerID="93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6" Nov 01 16:33:52.274524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.274507 4573 scope.go:122] "RemoveContainer" containerID="5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08" Nov 01 16:33:52.274612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.274584 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6"] Nov 01 16:33:52.274909 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:52.274880 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08\": container with ID starting with 5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08 not found: ID does not exist" containerID="5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08" Nov 01 16:33:52.274999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.274922 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08"} err="failed to get container status \"5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08\": rpc error: code = NotFound desc = could not find container \"5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08\": container with ID starting with 5949f1a9e03ecce7e6509b7a2449438e7f7c02eb11cd55460309abf1b054cb08 not found: ID does not exist" Nov 01 16:33:52.274999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.274947 4573 scope.go:122] "RemoveContainer" containerID="93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6" Nov 01 16:33:52.275279 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:52.275261 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6\": container with ID starting with 93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6 not found: ID does not exist" containerID="93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6" Nov 01 16:33:52.275370 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.275298 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6"} err="failed to get container status \"93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6\": rpc error: code = NotFound desc = could not find container \"93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6\": container with ID starting with 93f3c5b1ae9eea0850876726be0569a769112fd57d84158293f36f8064fed1f6 not found: ID does not exist" Nov 01 16:33:52.277865 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.277847 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f01de5-openshift-default-75bf69d85f-v6jb6"] Nov 01 16:33:52.458213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.458135 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f540387f-07a4-4601-b63a-99922263287f" path="/var/lib/kubelet/pods/f540387f-07a4-4601-b63a-99922263287f/volumes" Nov 01 16:33:52.970743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:52.970710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:53.101998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:53.101945 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4"] Nov 01 16:33:53.147527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:53.147495 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:53.248798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:53.248710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:53.897668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:53.897633 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc"] Nov 01 16:33:53.948017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:53.947948 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:54.048976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.048941 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:54.176686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.176594 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:54.364338 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:54.364305 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:33:54.726780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.726698 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:33:54.775177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.775154 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:33:54.874206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.874172 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:54.902526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.902498 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eba7eb6-54bb-45e1-9522-886df60e939f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba7eb6-54bb-45e1-9522-886df60e939f-plugins-config-volume\") pod \"5eba7eb6-54bb-45e1-9522-886df60e939f\" (UID: \"5eba7eb6-54bb-45e1-9522-886df60e939f\") " Nov 01 16:33:54.902960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.902542 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75996380-b13c-4d2a-8a70-494c0bf2ac06-kube-api-access-rwk22\" (UniqueName: \"kubernetes.io/projected/75996380-b13c-4d2a-8a70-494c0bf2ac06-kube-api-access-rwk22\") pod \"75996380-b13c-4d2a-8a70-494c0bf2ac06\" (UID: \"75996380-b13c-4d2a-8a70-494c0bf2ac06\") " Nov 01 16:33:54.902960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.902643 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eba7eb6-54bb-45e1-9522-886df60e939f-kube-api-access-85fsw\" (UniqueName: \"kubernetes.io/projected/5eba7eb6-54bb-45e1-9522-886df60e939f-kube-api-access-85fsw\") pod \"5eba7eb6-54bb-45e1-9522-886df60e939f\" (UID: \"5eba7eb6-54bb-45e1-9522-886df60e939f\") " Nov 01 16:33:54.902960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.902721 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75996380-b13c-4d2a-8a70-494c0bf2ac06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75996380-b13c-4d2a-8a70-494c0bf2ac06-plugins-config-volume\") pod \"75996380-b13c-4d2a-8a70-494c0bf2ac06\" (UID: \"75996380-b13c-4d2a-8a70-494c0bf2ac06\") " Nov 01 16:33:54.903213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.903065 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eba7eb6-54bb-45e1-9522-886df60e939f-plugins-config-volume" pod "5eba7eb6-54bb-45e1-9522-886df60e939f" (UID: "5eba7eb6-54bb-45e1-9522-886df60e939f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:54.903213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.903124 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75996380-b13c-4d2a-8a70-494c0bf2ac06-plugins-config-volume" pod "75996380-b13c-4d2a-8a70-494c0bf2ac06" (UID: "75996380-b13c-4d2a-8a70-494c0bf2ac06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:54.903321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.903288 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eba7eb6-54bb-45e1-9522-886df60e939f-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:54.903321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.903309 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75996380-b13c-4d2a-8a70-494c0bf2ac06-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:54.904853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.904829 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eba7eb6-54bb-45e1-9522-886df60e939f-kube-api-access-85fsw" pod "5eba7eb6-54bb-45e1-9522-886df60e939f" (UID: "5eba7eb6-54bb-45e1-9522-886df60e939f"). InnerVolumeSpecName "kube-api-access-85fsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:54.904924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:54.904849 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75996380-b13c-4d2a-8a70-494c0bf2ac06-kube-api-access-rwk22" pod "75996380-b13c-4d2a-8a70-494c0bf2ac06" (UID: "75996380-b13c-4d2a-8a70-494c0bf2ac06"). InnerVolumeSpecName "kube-api-access-rwk22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:55.004180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.004096 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85fsw\" (UniqueName: \"kubernetes.io/projected/5eba7eb6-54bb-45e1-9522-886df60e939f-kube-api-access-85fsw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:55.004180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.004127 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwk22\" (UniqueName: \"kubernetes.io/projected/75996380-b13c-4d2a-8a70-494c0bf2ac06-kube-api-access-rwk22\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:55.270486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.270399 4573 generic.go:356] "Generic (PLEG): container finished" podID="75996380-b13c-4d2a-8a70-494c0bf2ac06" containerID="0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170" exitCode=0 Nov 01 16:33:55.270486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.270447 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" Nov 01 16:33:55.270724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.270476 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" event={"ID":"75996380-b13c-4d2a-8a70-494c0bf2ac06","Type":"ContainerDied","Data":"0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170"} Nov 01 16:33:55.270724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.270530 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9" event={"ID":"75996380-b13c-4d2a-8a70-494c0bf2ac06","Type":"ContainerDied","Data":"24086ef591f0b1bc345a46514241c2e0b1aed58e615a6e9c67cafff7be7db8f3"} Nov 01 16:33:55.270724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.270548 4573 scope.go:122] "RemoveContainer" containerID="0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170" Nov 01 16:33:55.272002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.271979 4573 generic.go:356] "Generic (PLEG): container finished" podID="5eba7eb6-54bb-45e1-9522-886df60e939f" containerID="0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373" exitCode=0 Nov 01 16:33:55.272119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.272023 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" Nov 01 16:33:55.272119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.272068 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" event={"ID":"5eba7eb6-54bb-45e1-9522-886df60e939f","Type":"ContainerDied","Data":"0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373"} Nov 01 16:33:55.272247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.272125 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6" event={"ID":"5eba7eb6-54bb-45e1-9522-886df60e939f","Type":"ContainerDied","Data":"c1d182ee5623209cb018bd3caf0948fb1db54852e7abef4f2fb0d4b2139c16e1"} Nov 01 16:33:55.285338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.285325 4573 scope.go:122] "RemoveContainer" containerID="0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170" Nov 01 16:33:55.285663 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:55.285636 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170\": container with ID starting with 0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170 not found: ID does not exist" containerID="0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170" Nov 01 16:33:55.285784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.285672 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170"} err="failed to get container status \"0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170\": rpc error: code = NotFound desc = could not find container \"0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170\": container with ID starting with 0f6aa5b8ff93ebf4c847ee15568803b78b0eef5d6e3349d72645d5518cb14170 not found: ID does not exist" Nov 01 16:33:55.285784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.285694 4573 scope.go:122] "RemoveContainer" containerID="0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373" Nov 01 16:33:55.294710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.294694 4573 scope.go:122] "RemoveContainer" containerID="0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373" Nov 01 16:33:55.295008 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:55.294986 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373\": container with ID starting with 0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373 not found: ID does not exist" containerID="0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373" Nov 01 16:33:55.295066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.295018 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373"} err="failed to get container status \"0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373\": rpc error: code = NotFound desc = could not find container \"0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373\": container with ID starting with 0051ca7862adbb29038213929aff286cab363354f3308991de54c3ae238c4373 not found: ID does not exist" Nov 01 16:33:55.302157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.302133 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6"] Nov 01 16:33:55.306851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.306828 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-c8537c-dcfd4d4cf-qr8r6"] Nov 01 16:33:55.317069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.317049 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9"] Nov 01 16:33:55.322944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.322922 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-c8537c-8665fb9775-qkjd9"] Nov 01 16:33:55.719272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:55.719241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:56.220560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:56.220520 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:56.315068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:56.315029 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:56.473132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:56.472854 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eba7eb6-54bb-45e1-9522-886df60e939f" path="/var/lib/kubelet/pods/5eba7eb6-54bb-45e1-9522-886df60e939f/volumes" Nov 01 16:33:56.473688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:56.473664 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75996380-b13c-4d2a-8a70-494c0bf2ac06" path="/var/lib/kubelet/pods/75996380-b13c-4d2a-8a70-494c0bf2ac06/volumes" Nov 01 16:33:56.768589 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:56.768508 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:56.900009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:56.899976 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s"] Nov 01 16:33:56.954180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:56.954144 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:57.029009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:57.028931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:57.481921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:57.481885 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:58.102522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.102453 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" podUID="dad7ef24-f0a2-4901-8301-fb6e72ccfb81" containerName="payload-pre-processing" containerID="cri-o://c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763" gracePeriod=25 Nov 01 16:33:58.172713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.172678 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:58.286279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.286243 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:33:58.287763 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.287731 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:33:58.296810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.296784 4573 generic.go:356] "Generic (PLEG): container finished" podID="e416ad18-b9a1-4e75-8d3a-db6cf4a20203" containerID="a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157" exitCode=0 Nov 01 16:33:58.296944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.296819 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" Nov 01 16:33:58.296944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.296864 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" event={"ID":"e416ad18-b9a1-4e75-8d3a-db6cf4a20203","Type":"ContainerDied","Data":"a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157"} Nov 01 16:33:58.296944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.296913 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z" event={"ID":"e416ad18-b9a1-4e75-8d3a-db6cf4a20203","Type":"ContainerDied","Data":"efac1e7ce603989cc60e891ae4616bc3386b8d864ad717551407c4c971bb8200"} Nov 01 16:33:58.296944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.296940 4573 scope.go:122] "RemoveContainer" containerID="a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157" Nov 01 16:33:58.312606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.312584 4573 scope.go:122] "RemoveContainer" containerID="a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157" Nov 01 16:33:58.313024 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:58.312997 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157\": container with ID starting with a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157 not found: ID does not exist" containerID="a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157" Nov 01 16:33:58.313192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.313154 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157"} err="failed to get container status \"a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157\": rpc error: code = NotFound desc = could not find container \"a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157\": container with ID starting with a02636c107befa30c6ad30d2bdf59d74da29b357ac66efa238c7bf81795da157 not found: ID does not exist" Nov 01 16:33:58.442046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.442014 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-kube-api-access-v4mj6\" (UniqueName: \"kubernetes.io/projected/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-kube-api-access-v4mj6\") pod \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\" (UID: \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\") " Nov 01 16:33:58.442219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.442094 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-plugins-config-volume\") pod \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\" (UID: \"e416ad18-b9a1-4e75-8d3a-db6cf4a20203\") " Nov 01 16:33:58.442466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.442438 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-plugins-config-volume" pod "e416ad18-b9a1-4e75-8d3a-db6cf4a20203" (UID: "e416ad18-b9a1-4e75-8d3a-db6cf4a20203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:58.444122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.444101 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-kube-api-access-v4mj6" pod "e416ad18-b9a1-4e75-8d3a-db6cf4a20203" (UID: "e416ad18-b9a1-4e75-8d3a-db6cf4a20203"). InnerVolumeSpecName "kube-api-access-v4mj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:58.543661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.543587 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4mj6\" (UniqueName: \"kubernetes.io/projected/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-kube-api-access-v4mj6\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:58.543661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.543615 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e416ad18-b9a1-4e75-8d3a-db6cf4a20203-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:58.613984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.613940 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z"] Nov 01 16:33:58.616942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.616917 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f01de5-6cc545fd87-zxq6z"] Nov 01 16:33:58.692666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.692627 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j"] Nov 01 16:33:58.693202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693186 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" Nov 01 16:33:58.693262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693204 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" Nov 01 16:33:58.693262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693221 4573 cpu_manager.go:414] "containerMap: removing container" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" Nov 01 16:33:58.693262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693227 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" Nov 01 16:33:58.693262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693244 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e416ad18-b9a1-4e75-8d3a-db6cf4a20203" containerName="payload-pre-processing" Nov 01 16:33:58.693262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693252 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e416ad18-b9a1-4e75-8d3a-db6cf4a20203" containerName="payload-pre-processing" Nov 01 16:33:58.693262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693263 4573 cpu_manager.go:414] "containerMap: removing container" podUID="75996380-b13c-4d2a-8a70-494c0bf2ac06" containerName="payload-processing" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693268 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75996380-b13c-4d2a-8a70-494c0bf2ac06" containerName="payload-processing" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693291 4573 cpu_manager.go:414] "containerMap: removing container" podUID="245b7454-2574-408f-bf01-8dd608326607" containerName="revoke-keys" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693296 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245b7454-2574-408f-bf01-8dd608326607" containerName="revoke-keys" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693312 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5eba7eb6-54bb-45e1-9522-886df60e939f" containerName="payload-pre-processing" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693316 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eba7eb6-54bb-45e1-9522-886df60e939f" containerName="payload-pre-processing" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693397 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="245b7454-2574-408f-bf01-8dd608326607" containerName="revoke-keys" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693409 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eba7eb6-54bb-45e1-9522-886df60e939f" containerName="payload-pre-processing" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693417 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693423 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e416ad18-b9a1-4e75-8d3a-db6cf4a20203" containerName="payload-pre-processing" Nov 01 16:33:58.693464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.693429 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="75996380-b13c-4d2a-8a70-494c0bf2ac06" containerName="payload-processing" Nov 01 16:33:58.698074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.698054 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx"] Nov 01 16:33:58.698194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.698174 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.698692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.698676 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="f540387f-07a4-4601-b63a-99922263287f" containerName="istio-proxy" Nov 01 16:33:58.702159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.702138 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.703419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.703389 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:33:58.704313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.704276 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:33:58.704825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.704743 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j"] Nov 01 16:33:58.709789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.708934 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx"] Nov 01 16:33:58.847048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.846967 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9jk2\" (UniqueName: \"kubernetes.io/projected/8380742e-e617-4116-847a-2427d0841ae5-kube-api-access-k9jk2\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.847048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.847003 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjp8z\" (UniqueName: \"kubernetes.io/projected/589709e8-9c48-49f5-91e4-d4717c7e3b42-kube-api-access-xjp8z\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.847251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.847119 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380742e-e617-4116-847a-2427d0841ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.847293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.847267 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.847345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.847331 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380742e-e617-4116-847a-2427d0841ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.847387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.847376 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.898107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.898065 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" podUID="96a8bdf5-8654-4e4f-a2c7-3974af866440" containerName="payload-pre-processing" containerID="cri-o://f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee" gracePeriod=25 Nov 01 16:33:58.948225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.948196 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380742e-e617-4116-847a-2427d0841ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.948359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.948236 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.948359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.948268 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9jk2\" (UniqueName: \"kubernetes.io/projected/8380742e-e617-4116-847a-2427d0841ae5-kube-api-access-k9jk2\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.948359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.948296 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjp8z\" (UniqueName: \"kubernetes.io/projected/589709e8-9c48-49f5-91e4-d4717c7e3b42-kube-api-access-xjp8z\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.948539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.948372 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380742e-e617-4116-847a-2427d0841ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.948539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.948527 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.949054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.949023 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380742e-e617-4116-847a-2427d0841ae5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.949194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.949177 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.950592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.950563 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.950662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.950613 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380742e-e617-4116-847a-2427d0841ae5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.955397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.955378 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9jk2\" (UniqueName: \"kubernetes.io/projected/8380742e-e617-4116-847a-2427d0841ae5-kube-api-access-k9jk2\") pod \"maas-api-revoke-sub-e28a7d65-czrvx\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:58.955484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.955430 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjp8z\" (UniqueName: \"kubernetes.io/projected/589709e8-9c48-49f5-91e4-d4717c7e3b42-kube-api-access-xjp8z\") pod \"maas-api-revoke-sub-5ff31524-xhf4j\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:58.965415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:58.965391 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:59.021478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.021310 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:33:59.039856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.026059 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:33:59.205247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.205223 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:59.305481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.305450 4573 generic.go:356] "Generic (PLEG): container finished" podID="96a8bdf5-8654-4e4f-a2c7-3974af866440" containerID="f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee" exitCode=0 Nov 01 16:33:59.305634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.305482 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" event={"ID":"96a8bdf5-8654-4e4f-a2c7-3974af866440","Type":"ContainerDied","Data":"f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee"} Nov 01 16:33:59.305634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.305528 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" Nov 01 16:33:59.305634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.305537 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc" event={"ID":"96a8bdf5-8654-4e4f-a2c7-3974af866440","Type":"ContainerDied","Data":"fb60830b5285396972c3bc879ca0850df1d63e11487ccf335130ece32fc24fd2"} Nov 01 16:33:59.305634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.305563 4573 scope.go:122] "RemoveContainer" containerID="f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee" Nov 01 16:33:59.316872 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.316851 4573 scope.go:122] "RemoveContainer" containerID="f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee" Nov 01 16:33:59.317160 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:33:59.317141 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee\": container with ID starting with f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee not found: ID does not exist" containerID="f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee" Nov 01 16:33:59.317239 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.317167 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee"} err="failed to get container status \"f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee\": rpc error: code = NotFound desc = could not find container \"f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee\": container with ID starting with f6daa71fdd1c2a142531593f08e6448c678a24d623d2bcf526f24629192fe7ee not found: ID does not exist" Nov 01 16:33:59.333805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.333147 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j"] Nov 01 16:33:59.336286 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:33:59.336259 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589709e8_9c48_49f5_91e4_d4717c7e3b42.slice/crio-7c5ad1331099435fdd5dc15cf3ddbef1221814653948c1182f73e86dead90dcb WatchSource:0}: Error finding container 7c5ad1331099435fdd5dc15cf3ddbef1221814653948c1182f73e86dead90dcb: Status 404 returned error can't find the container with id 7c5ad1331099435fdd5dc15cf3ddbef1221814653948c1182f73e86dead90dcb Nov 01 16:33:59.355507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.355474 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a8bdf5-8654-4e4f-a2c7-3974af866440-kube-api-access-624zx\" (UniqueName: \"kubernetes.io/projected/96a8bdf5-8654-4e4f-a2c7-3974af866440-kube-api-access-624zx\") pod \"96a8bdf5-8654-4e4f-a2c7-3974af866440\" (UID: \"96a8bdf5-8654-4e4f-a2c7-3974af866440\") " Nov 01 16:33:59.355627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.355531 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a8bdf5-8654-4e4f-a2c7-3974af866440-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a8bdf5-8654-4e4f-a2c7-3974af866440-plugins-config-volume\") pod \"96a8bdf5-8654-4e4f-a2c7-3974af866440\" (UID: \"96a8bdf5-8654-4e4f-a2c7-3974af866440\") " Nov 01 16:33:59.356547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.356506 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx"] Nov 01 16:33:59.356651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.356576 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a8bdf5-8654-4e4f-a2c7-3974af866440-plugins-config-volume" pod "96a8bdf5-8654-4e4f-a2c7-3974af866440" (UID: "96a8bdf5-8654-4e4f-a2c7-3974af866440"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:33:59.364638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.364614 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a8bdf5-8654-4e4f-a2c7-3974af866440-kube-api-access-624zx" pod "96a8bdf5-8654-4e4f-a2c7-3974af866440" (UID: "96a8bdf5-8654-4e4f-a2c7-3974af866440"). InnerVolumeSpecName "kube-api-access-624zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:33:59.456864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.456833 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-624zx\" (UniqueName: \"kubernetes.io/projected/96a8bdf5-8654-4e4f-a2c7-3974af866440-kube-api-access-624zx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:59.456864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.456860 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96a8bdf5-8654-4e4f-a2c7-3974af866440-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:33:59.615520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.615439 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h"] Nov 01 16:33:59.616103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.616056 4573 cpu_manager.go:414] "containerMap: removing container" podUID="96a8bdf5-8654-4e4f-a2c7-3974af866440" containerName="payload-pre-processing" Nov 01 16:33:59.616103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.616072 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a8bdf5-8654-4e4f-a2c7-3974af866440" containerName="payload-pre-processing" Nov 01 16:33:59.616234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.616193 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a8bdf5-8654-4e4f-a2c7-3974af866440" containerName="payload-pre-processing" Nov 01 16:33:59.620942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.620922 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.628297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.628271 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h"] Nov 01 16:33:59.642458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.642432 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc"] Nov 01 16:33:59.647340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.647315 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f4a265-6c7585d45f-zmhsc"] Nov 01 16:33:59.760566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.760535 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.760765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.760579 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.760765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.760723 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2c2w\" (UniqueName: \"kubernetes.io/projected/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-kube-api-access-m2c2w\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.861633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.861599 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.861882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.861647 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.861882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.861713 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2c2w\" (UniqueName: \"kubernetes.io/projected/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-kube-api-access-m2c2w\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.862367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.862337 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.864161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.864131 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.871280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.871229 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2c2w\" (UniqueName: \"kubernetes.io/projected/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-kube-api-access-m2c2w\") pod \"maas-api-revoke-sub-e942cb9b-q7z8h\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:33:59.876453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.876433 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:33:59.939003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:33:59.938969 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:34:00.218252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.218219 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h"] Nov 01 16:34:00.232668 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:34:00.232633 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e04db5_df2a_4b58_b21d_68c2d6bc0ef4.slice/crio-2e58f6abeb6d5616218733422396e28badb1215b65d03d27b847a07892e9c806 WatchSource:0}: Error finding container 2e58f6abeb6d5616218733422396e28badb1215b65d03d27b847a07892e9c806: Status 404 returned error can't find the container with id 2e58f6abeb6d5616218733422396e28badb1215b65d03d27b847a07892e9c806 Nov 01 16:34:00.315366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.315335 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" event={"ID":"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4","Type":"ContainerStarted","Data":"2e58f6abeb6d5616218733422396e28badb1215b65d03d27b847a07892e9c806"} Nov 01 16:34:00.317030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.317000 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" event={"ID":"589709e8-9c48-49f5-91e4-d4717c7e3b42","Type":"ContainerStarted","Data":"188ab1fa1f7cfb6e5a632b3265a4bcd5cdb866f1bb78643ccddb776605c4b9bb"} Nov 01 16:34:00.317165 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.317050 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" event={"ID":"589709e8-9c48-49f5-91e4-d4717c7e3b42","Type":"ContainerStarted","Data":"7c5ad1331099435fdd5dc15cf3ddbef1221814653948c1182f73e86dead90dcb"} Nov 01 16:34:00.318697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.318673 4573 generic.go:356] "Generic (PLEG): container finished" podID="8380742e-e617-4116-847a-2427d0841ae5" containerID="db195488edac1931324e5f21e5778c8b8a32bef95eadc8fd2779f0de7de817bd" exitCode=0 Nov 01 16:34:00.318826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.318784 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" event={"ID":"8380742e-e617-4116-847a-2427d0841ae5","Type":"ContainerDied","Data":"db195488edac1931324e5f21e5778c8b8a32bef95eadc8fd2779f0de7de817bd"} Nov 01 16:34:00.318870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.318830 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" event={"ID":"8380742e-e617-4116-847a-2427d0841ae5","Type":"ContainerStarted","Data":"e08a02fb0c089af7215143c312ee7d0c42968af6a773e5aed3c35b4408c5cc45"} Nov 01 16:34:00.335268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.335219 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" podStartSLOduration=2.3352057840000002 podStartE2EDuration="2.335205784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:34:00.332319959 +0000 UTC m=+2778.035485896" watchObservedRunningTime="2026-11-01 16:34:00.335205784 +0000 UTC m=+2778.038371713" Nov 01 16:34:00.459251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.459186 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a8bdf5-8654-4e4f-a2c7-3974af866440" path="/var/lib/kubelet/pods/96a8bdf5-8654-4e4f-a2c7-3974af866440/volumes" Nov 01 16:34:00.459594 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.459581 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e416ad18-b9a1-4e75-8d3a-db6cf4a20203" path="/var/lib/kubelet/pods/e416ad18-b9a1-4e75-8d3a-db6cf4a20203/volumes" Nov 01 16:34:00.582156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:00.582131 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:01.328073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.328039 4573 generic.go:356] "Generic (PLEG): container finished" podID="589709e8-9c48-49f5-91e4-d4717c7e3b42" containerID="188ab1fa1f7cfb6e5a632b3265a4bcd5cdb866f1bb78643ccddb776605c4b9bb" exitCode=0 Nov 01 16:34:01.328531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.328129 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" event={"ID":"589709e8-9c48-49f5-91e4-d4717c7e3b42","Type":"ContainerDied","Data":"188ab1fa1f7cfb6e5a632b3265a4bcd5cdb866f1bb78643ccddb776605c4b9bb"} Nov 01 16:34:01.329998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.329967 4573 generic.go:356] "Generic (PLEG): container finished" podID="c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4" containerID="d9c8d44656ab9942b560d664646beb6e085ddcf037dce5f97dc1d09ec3225b84" exitCode=0 Nov 01 16:34:01.330129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.330055 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" event={"ID":"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4","Type":"ContainerDied","Data":"d9c8d44656ab9942b560d664646beb6e085ddcf037dce5f97dc1d09ec3225b84"} Nov 01 16:34:01.429049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.428936 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:01.474802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.474775 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:34:01.485565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.485524 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8380742e-e617-4116-847a-2427d0841ae5-kube-api-access-k9jk2\" (UniqueName: \"kubernetes.io/projected/8380742e-e617-4116-847a-2427d0841ae5-kube-api-access-k9jk2\") pod \"8380742e-e617-4116-847a-2427d0841ae5\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " Nov 01 16:34:01.485808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.485600 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8380742e-e617-4116-847a-2427d0841ae5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380742e-e617-4116-847a-2427d0841ae5-maas-api-xks-ca\") pod \"8380742e-e617-4116-847a-2427d0841ae5\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " Nov 01 16:34:01.485808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.485768 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8380742e-e617-4116-847a-2427d0841ae5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380742e-e617-4116-847a-2427d0841ae5-maas-api-openshift-ca\") pod \"8380742e-e617-4116-847a-2427d0841ae5\" (UID: \"8380742e-e617-4116-847a-2427d0841ae5\") " Nov 01 16:34:01.487043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.486959 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8380742e-e617-4116-847a-2427d0841ae5-maas-api-openshift-ca" pod "8380742e-e617-4116-847a-2427d0841ae5" (UID: "8380742e-e617-4116-847a-2427d0841ae5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:01.490834 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.490807 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8380742e-e617-4116-847a-2427d0841ae5-kube-api-access-k9jk2" pod "8380742e-e617-4116-847a-2427d0841ae5" (UID: "8380742e-e617-4116-847a-2427d0841ae5"). InnerVolumeSpecName "kube-api-access-k9jk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:01.500267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.497910 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8380742e-e617-4116-847a-2427d0841ae5-maas-api-xks-ca" pod "8380742e-e617-4116-847a-2427d0841ae5" (UID: "8380742e-e617-4116-847a-2427d0841ae5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:01.587581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.587547 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9jk2\" (UniqueName: \"kubernetes.io/projected/8380742e-e617-4116-847a-2427d0841ae5-kube-api-access-k9jk2\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:01.587581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.587576 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8380742e-e617-4116-847a-2427d0841ae5-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:01.587581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.587586 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8380742e-e617-4116-847a-2427d0841ae5-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:01.901082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:01.901022 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" podUID="1676214c-b5c3-42ae-a139-e6f0098f23cd" containerName="payload-pre-processing" containerID="cri-o://5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23" gracePeriod=25 Nov 01 16:34:02.167649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.167516 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:02.267642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.267581 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:02.332962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.332923 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" podUID="dad7ef24-f0a2-4901-8301-fb6e72ccfb81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:34:02.339002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.338983 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" Nov 01 16:34:02.339119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.339018 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx" event={"ID":"8380742e-e617-4116-847a-2427d0841ae5","Type":"ContainerDied","Data":"e08a02fb0c089af7215143c312ee7d0c42968af6a773e5aed3c35b4408c5cc45"} Nov 01 16:34:02.339119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.339065 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e08a02fb0c089af7215143c312ee7d0c42968af6a773e5aed3c35b4408c5cc45" Nov 01 16:34:02.484902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.484876 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:34:02.498257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.498233 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-openshift-ca\") pod \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " Nov 01 16:34:02.498373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.498303 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-kube-api-access-m2c2w\" (UniqueName: \"kubernetes.io/projected/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-kube-api-access-m2c2w\") pod \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " Nov 01 16:34:02.498458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.498405 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-xks-ca\") pod \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\" (UID: \"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4\") " Nov 01 16:34:02.498648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.498623 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-openshift-ca" pod "c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4" (UID: "c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:02.498926 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.498904 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.501044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.500857 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-xks-ca" pod "c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4" (UID: "c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:02.501044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.501020 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-kube-api-access-m2c2w" pod "c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4" (UID: "c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4"). InnerVolumeSpecName "kube-api-access-m2c2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:02.524018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.523993 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:34:02.599843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.599808 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-openshift-ca\") pod \"589709e8-9c48-49f5-91e4-d4717c7e3b42\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " Nov 01 16:34:02.599993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.599919 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589709e8-9c48-49f5-91e4-d4717c7e3b42-kube-api-access-xjp8z\" (UniqueName: \"kubernetes.io/projected/589709e8-9c48-49f5-91e4-d4717c7e3b42-kube-api-access-xjp8z\") pod \"589709e8-9c48-49f5-91e4-d4717c7e3b42\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " Nov 01 16:34:02.600033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.600020 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-xks-ca\") pod \"589709e8-9c48-49f5-91e4-d4717c7e3b42\" (UID: \"589709e8-9c48-49f5-91e4-d4717c7e3b42\") " Nov 01 16:34:02.600180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.600157 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-openshift-ca" pod "589709e8-9c48-49f5-91e4-d4717c7e3b42" (UID: "589709e8-9c48-49f5-91e4-d4717c7e3b42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:02.600413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.600398 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.600487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.600416 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.600487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.600427 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2c2w\" (UniqueName: \"kubernetes.io/projected/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4-kube-api-access-m2c2w\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.601943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.601923 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589709e8-9c48-49f5-91e4-d4717c7e3b42-kube-api-access-xjp8z" pod "589709e8-9c48-49f5-91e4-d4717c7e3b42" (UID: "589709e8-9c48-49f5-91e4-d4717c7e3b42"). InnerVolumeSpecName "kube-api-access-xjp8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:02.602035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.602007 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-xks-ca" pod "589709e8-9c48-49f5-91e4-d4717c7e3b42" (UID: "589709e8-9c48-49f5-91e4-d4717c7e3b42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:02.682118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.682075 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" containerID="cri-o://095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e" gracePeriod=45 Nov 01 16:34:02.701399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.701366 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjp8z\" (UniqueName: \"kubernetes.io/projected/589709e8-9c48-49f5-91e4-d4717c7e3b42-kube-api-access-xjp8z\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.701399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.701397 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/589709e8-9c48-49f5-91e4-d4717c7e3b42-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:02.862093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.862047 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:02.935401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:02.935367 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:34:03.004101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004070 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-tmp-dir\") pod \"af872790-62d6-41e2-be43-07c7511b047b\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " Nov 01 16:34:03.004263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004176 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-model-cache\") pod \"af872790-62d6-41e2-be43-07c7511b047b\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " Nov 01 16:34:03.004263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004198 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af872790-62d6-41e2-be43-07c7511b047b-kube-api-access-t5kdf\" (UniqueName: \"kubernetes.io/projected/af872790-62d6-41e2-be43-07c7511b047b-kube-api-access-t5kdf\") pod \"af872790-62d6-41e2-be43-07c7511b047b\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " Nov 01 16:34:03.004263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004223 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af872790-62d6-41e2-be43-07c7511b047b-tls-certs\" (UniqueName: \"kubernetes.io/secret/af872790-62d6-41e2-be43-07c7511b047b-tls-certs\") pod \"af872790-62d6-41e2-be43-07c7511b047b\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " Nov 01 16:34:03.004423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004279 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-home\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-home\") pod \"af872790-62d6-41e2-be43-07c7511b047b\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " Nov 01 16:34:03.004423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004330 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-dshm\") pod \"af872790-62d6-41e2-be43-07c7511b047b\" (UID: \"af872790-62d6-41e2-be43-07c7511b047b\") " Nov 01 16:34:03.004526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004422 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-tmp-dir" pod "af872790-62d6-41e2-be43-07c7511b047b" (UID: "af872790-62d6-41e2-be43-07c7511b047b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.004595 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004517 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-model-cache" pod "af872790-62d6-41e2-be43-07c7511b047b" (UID: "af872790-62d6-41e2-be43-07c7511b047b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.004708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004679 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-home" pod "af872790-62d6-41e2-be43-07c7511b047b" (UID: "af872790-62d6-41e2-be43-07c7511b047b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.004864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004842 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.004945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004866 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.004945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.004882 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.006571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.006545 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-dshm" pod "af872790-62d6-41e2-be43-07c7511b047b" (UID: "af872790-62d6-41e2-be43-07c7511b047b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:03.006644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.006596 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af872790-62d6-41e2-be43-07c7511b047b-tls-certs" pod "af872790-62d6-41e2-be43-07c7511b047b" (UID: "af872790-62d6-41e2-be43-07c7511b047b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:03.007085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.007059 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af872790-62d6-41e2-be43-07c7511b047b-kube-api-access-t5kdf" pod "af872790-62d6-41e2-be43-07c7511b047b" (UID: "af872790-62d6-41e2-be43-07c7511b047b"). InnerVolumeSpecName "kube-api-access-t5kdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:03.106723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.106686 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5kdf\" (UniqueName: \"kubernetes.io/projected/af872790-62d6-41e2-be43-07c7511b047b-kube-api-access-t5kdf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.106723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.106720 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af872790-62d6-41e2-be43-07c7511b047b-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.106995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.106731 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af872790-62d6-41e2-be43-07c7511b047b-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:03.347453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.347427 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" Nov 01 16:34:03.347907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.347467 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h" event={"ID":"c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4","Type":"ContainerDied","Data":"2e58f6abeb6d5616218733422396e28badb1215b65d03d27b847a07892e9c806"} Nov 01 16:34:03.347907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.347526 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e58f6abeb6d5616218733422396e28badb1215b65d03d27b847a07892e9c806" Nov 01 16:34:03.348982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.348962 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" Nov 01 16:34:03.349104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.349008 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j" event={"ID":"589709e8-9c48-49f5-91e4-d4717c7e3b42","Type":"ContainerDied","Data":"7c5ad1331099435fdd5dc15cf3ddbef1221814653948c1182f73e86dead90dcb"} Nov 01 16:34:03.349104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.349049 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c5ad1331099435fdd5dc15cf3ddbef1221814653948c1182f73e86dead90dcb" Nov 01 16:34:03.350570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.350550 4573 generic.go:356] "Generic (PLEG): container finished" podID="af872790-62d6-41e2-be43-07c7511b047b" containerID="095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e" exitCode=0 Nov 01 16:34:03.350686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.350592 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" Nov 01 16:34:03.350686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.350639 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" event={"ID":"af872790-62d6-41e2-be43-07c7511b047b","Type":"ContainerDied","Data":"095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e"} Nov 01 16:34:03.350823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.350686 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56" event={"ID":"af872790-62d6-41e2-be43-07c7511b047b","Type":"ContainerDied","Data":"39138a47ff0af2806ea31d3f9d0a6705b10deaa9373bc0904d68eefb2f75b9ab"} Nov 01 16:34:03.350823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.350709 4573 scope.go:122] "RemoveContainer" containerID="095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e" Nov 01 16:34:03.369220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.368960 4573 scope.go:122] "RemoveContainer" containerID="095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e" Nov 01 16:34:03.369455 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:03.369422 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e\": container with ID starting with 095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e not found: ID does not exist" containerID="095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e" Nov 01 16:34:03.369522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.369465 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e"} err="failed to get container status \"095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e\": rpc error: code = NotFound desc = could not find container \"095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e\": container with ID starting with 095e5bb474bf561623cf9b030a6478c44a27c4f12229346e8e8ba5f2287a587e not found: ID does not exist" Nov 01 16:34:03.386845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.386820 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56"] Nov 01 16:34:03.392728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.392700 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-dfff59/distinct-w1-dfff59-kserve-56fccf5f84-cts56"] Nov 01 16:34:03.685700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:03.685613 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:04.457792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:04.457738 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af872790-62d6-41e2-be43-07c7511b047b" path="/var/lib/kubelet/pods/af872790-62d6-41e2-be43-07c7511b047b/volumes" Nov 01 16:34:04.464623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:04.464597 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:05.186266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:05.186232 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:05.355194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:05.355154 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" podUID="1676214c-b5c3-42ae-a139-e6f0098f23cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:34:06.187544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.187511 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:06.639916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.639883 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" containerID="cri-o://8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580" gracePeriod=45 Nov 01 16:34:06.910998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.910974 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:34:06.953290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.953253 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-home\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-home\") pod \"839701ef-516c-4451-b096-50ca25cdcabc\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " Nov 01 16:34:06.953473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.953356 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-tmp-dir\") pod \"839701ef-516c-4451-b096-50ca25cdcabc\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " Nov 01 16:34:06.953473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.953395 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-dshm\") pod \"839701ef-516c-4451-b096-50ca25cdcabc\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " Nov 01 16:34:06.953473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.953433 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839701ef-516c-4451-b096-50ca25cdcabc-kube-api-access-h6b94\" (UniqueName: \"kubernetes.io/projected/839701ef-516c-4451-b096-50ca25cdcabc-kube-api-access-h6b94\") pod \"839701ef-516c-4451-b096-50ca25cdcabc\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " Nov 01 16:34:06.953636 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.953569 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/839701ef-516c-4451-b096-50ca25cdcabc-tls-certs\" (UniqueName: \"kubernetes.io/secret/839701ef-516c-4451-b096-50ca25cdcabc-tls-certs\") pod \"839701ef-516c-4451-b096-50ca25cdcabc\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " Nov 01 16:34:06.953689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.953651 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-model-cache\") pod \"839701ef-516c-4451-b096-50ca25cdcabc\" (UID: \"839701ef-516c-4451-b096-50ca25cdcabc\") " Nov 01 16:34:06.953784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.953733 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-tmp-dir" pod "839701ef-516c-4451-b096-50ca25cdcabc" (UID: "839701ef-516c-4451-b096-50ca25cdcabc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:06.954127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.954102 4573 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-tmp-dir\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:06.954127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.954117 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-home" pod "839701ef-516c-4451-b096-50ca25cdcabc" (UID: "839701ef-516c-4451-b096-50ca25cdcabc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:06.954375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.954353 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-model-cache" pod "839701ef-516c-4451-b096-50ca25cdcabc" (UID: "839701ef-516c-4451-b096-50ca25cdcabc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:06.955934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.955901 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-dshm" pod "839701ef-516c-4451-b096-50ca25cdcabc" (UID: "839701ef-516c-4451-b096-50ca25cdcabc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:06.956054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.956030 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839701ef-516c-4451-b096-50ca25cdcabc-kube-api-access-h6b94" pod "839701ef-516c-4451-b096-50ca25cdcabc" (UID: "839701ef-516c-4451-b096-50ca25cdcabc"). InnerVolumeSpecName "kube-api-access-h6b94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:06.956572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:06.956555 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/839701ef-516c-4451-b096-50ca25cdcabc-tls-certs" pod "839701ef-516c-4451-b096-50ca25cdcabc" (UID: "839701ef-516c-4451-b096-50ca25cdcabc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:07.055352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.055318 4573 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/839701ef-516c-4451-b096-50ca25cdcabc-tls-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.055352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.055351 4573 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-model-cache\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.055565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.055365 4573 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-home\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.055565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.055377 4573 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/839701ef-516c-4451-b096-50ca25cdcabc-dshm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.055565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.055389 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6b94\" (UniqueName: \"kubernetes.io/projected/839701ef-516c-4451-b096-50ca25cdcabc-kube-api-access-h6b94\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:07.067413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.067381 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56"] Nov 01 16:34:07.075151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.075123 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c238f07-8ml56"] Nov 01 16:34:07.163953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.163875 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:07.380974 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.380943 4573 generic.go:356] "Generic (PLEG): container finished" podID="839701ef-516c-4451-b096-50ca25cdcabc" containerID="8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580" exitCode=0 Nov 01 16:34:07.381407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.380989 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" Nov 01 16:34:07.381407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.381027 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" event={"ID":"839701ef-516c-4451-b096-50ca25cdcabc","Type":"ContainerDied","Data":"8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580"} Nov 01 16:34:07.381407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.381073 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg" event={"ID":"839701ef-516c-4451-b096-50ca25cdcabc","Type":"ContainerDied","Data":"e079d483ea5570569ae126c811a5d02763a0ef61043d088014ad0ddd21e4544e"} Nov 01 16:34:07.381407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.381096 4573 scope.go:122] "RemoveContainer" containerID="8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580" Nov 01 16:34:07.392160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.392143 4573 scope.go:122] "RemoveContainer" containerID="8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580" Nov 01 16:34:07.392436 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:07.392414 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580\": container with ID starting with 8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580 not found: ID does not exist" containerID="8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580" Nov 01 16:34:07.392524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.392446 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580"} err="failed to get container status \"8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580\": rpc error: code = NotFound desc = could not find container \"8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580\": container with ID starting with 8a2f71a15fe1df848950e045540d431d5a05a8be86447d1f49cbd6f19b33d580 not found: ID does not exist" Nov 01 16:34:07.404898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.404873 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg"] Nov 01 16:34:07.414550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.414495 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9f4dc9/distinct-2-w4-9f4dc9-kserve-7d8b959bb9-b84kg"] Nov 01 16:34:07.973327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:07.973289 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:08.027330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:08.027275 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222"] Nov 01 16:34:08.033377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:08.033344 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35da1e14-f5222"] Nov 01 16:34:08.457064 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:08.457031 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839701ef-516c-4451-b096-50ca25cdcabc" path="/var/lib/kubelet/pods/839701ef-516c-4451-b096-50ca25cdcabc/volumes" Nov 01 16:34:08.457479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:08.457465 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8927f4c4-a345-48c1-bbb7-c231c39c4952" path="/var/lib/kubelet/pods/8927f4c4-a345-48c1-bbb7-c231c39c4952/volumes" Nov 01 16:34:08.457862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:08.457849 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab785806-a92a-49bb-a609-653301fc203a" path="/var/lib/kubelet/pods/ab785806-a92a-49bb-a609-653301fc203a/volumes" Nov 01 16:34:08.811360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:08.811274 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:09.533195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:09.533167 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:09.744182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:09.744144 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:09.927997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:09.927960 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp"] Nov 01 16:34:09.932710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:09.932686 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-dfff59-swhzp"] Nov 01 16:34:09.933689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:09.933666 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:10.007041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:10.007008 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:10.383442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:10.383409 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:10.457153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:10.457117 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c4acb5f-73ff-4468-926d-29ab31b2857f" path="/var/lib/kubelet/pods/4c4acb5f-73ff-4468-926d-29ab31b2857f/volumes" Nov 01 16:34:10.654791 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:10.654672 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:10.833471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:10.826366 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:10.957175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:10.957094 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:10.990268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:10.990175 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7"] Nov 01 16:34:10.993642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:10.993622 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9f4dc9-g74k7"] Nov 01 16:34:11.061385 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:11.061349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:11.452934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:11.452897 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:12.331648 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:12.331569 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:12.341728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:12.341679 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" podUID="dad7ef24-f0a2-4901-8301-fb6e72ccfb81" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:34:12.457512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:12.457479 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245b7454-2574-408f-bf01-8dd608326607" path="/var/lib/kubelet/pods/245b7454-2574-408f-bf01-8dd608326607/volumes" Nov 01 16:34:13.146094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:13.146058 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:13.918949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:13.918919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:14.348535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:14.348435 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:34:14.787569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:14.787534 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:15.355715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:15.355672 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" podUID="1676214c-b5c3-42ae-a139-e6f0098f23cd" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:34:15.749369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:15.749326 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:16.560946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:16.560907 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:17.131415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:17.131375 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:17.229074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:17.229039 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:17.409816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:17.409692 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22"] Nov 01 16:34:17.410013 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:17.409969 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" containerID="cri-o://b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae" gracePeriod=30 Nov 01 16:34:17.433019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:17.432990 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:17.531254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:17.531218 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:18.360564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.360481 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 16:34:18.380795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.380735 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:18.470951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.470921 4573 generic.go:356] "Generic (PLEG): container finished" podID="dad7ef24-f0a2-4901-8301-fb6e72ccfb81" containerID="c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763" exitCode=0 Nov 01 16:34:18.471196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.470975 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" Nov 01 16:34:18.471196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.471009 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" event={"ID":"dad7ef24-f0a2-4901-8301-fb6e72ccfb81","Type":"ContainerDied","Data":"c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763"} Nov 01 16:34:18.471196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.471059 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4" event={"ID":"dad7ef24-f0a2-4901-8301-fb6e72ccfb81","Type":"ContainerDied","Data":"f177124e6e609c4a7c3fb12552a708cd95c17aed8a46a3230f55b12a32335953"} Nov 01 16:34:18.471196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.471083 4573 scope.go:122] "RemoveContainer" containerID="c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763" Nov 01 16:34:18.480793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.480775 4573 scope.go:122] "RemoveContainer" containerID="c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763" Nov 01 16:34:18.481077 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:18.481056 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763\": container with ID starting with c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763 not found: ID does not exist" containerID="c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763" Nov 01 16:34:18.481137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.481086 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763"} err="failed to get container status \"c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763\": rpc error: code = NotFound desc = could not find container \"c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763\": container with ID starting with c4dc352d4474ce2195a6d87f7369ae8624d24da28dae72d63b2d464b58dc2763 not found: ID does not exist" Nov 01 16:34:18.492501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.492481 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-kube-api-access-gjbc8\" (UniqueName: \"kubernetes.io/projected/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-kube-api-access-gjbc8\") pod \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\" (UID: \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\") " Nov 01 16:34:18.492575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.492550 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-plugins-config-volume\") pod \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\" (UID: \"dad7ef24-f0a2-4901-8301-fb6e72ccfb81\") " Nov 01 16:34:18.492936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.492914 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-plugins-config-volume" pod "dad7ef24-f0a2-4901-8301-fb6e72ccfb81" (UID: "dad7ef24-f0a2-4901-8301-fb6e72ccfb81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:18.494619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.494603 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-kube-api-access-gjbc8" pod "dad7ef24-f0a2-4901-8301-fb6e72ccfb81" (UID: "dad7ef24-f0a2-4901-8301-fb6e72ccfb81"). InnerVolumeSpecName "kube-api-access-gjbc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:18.594056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.594024 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:18.594056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.594049 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjbc8\" (UniqueName: \"kubernetes.io/projected/dad7ef24-f0a2-4901-8301-fb6e72ccfb81-kube-api-access-gjbc8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:18.798729 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.798701 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4"] Nov 01 16:34:18.804424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.804400 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-dfff59-749f5d5d6c-tc5d4"] Nov 01 16:34:18.893004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.892944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:18.993393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:18.993361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:19.032651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:19.032547 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b"] Nov 01 16:34:19.036794 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:19.036763 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b806f5b8-gnk7b"] Nov 01 16:34:19.093182 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:19.093139 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:19.184243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:19.184205 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k"] Nov 01 16:34:19.184534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:19.184505 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" containerID="cri-o://30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a" gracePeriod=30 Nov 01 16:34:19.213819 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:19.213786 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:19.780868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:19.780838 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:20.458465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:20.458433 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9109b78-94ed-4e86-a286-2a488fd0f3d0" path="/var/lib/kubelet/pods/a9109b78-94ed-4e86-a286-2a488fd0f3d0/volumes" Nov 01 16:34:20.458864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:20.458849 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dad7ef24-f0a2-4901-8301-fb6e72ccfb81" path="/var/lib/kubelet/pods/dad7ef24-f0a2-4901-8301-fb6e72ccfb81/volumes" Nov 01 16:34:20.769706 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:20.769625 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:21.411267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:21.411233 4573 patch_prober.go:29] interesting pod/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:34:21.411633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:21.411284 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:34:21.462720 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:21.462690 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:21.672876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:21.672794 4573 patch_prober.go:29] interesting pod/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:34:21.672876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:21.672853 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:34:22.170792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.170767 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 16:34:22.263036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.262997 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:22.341631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.341551 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1676214c-b5c3-42ae-a139-e6f0098f23cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676214c-b5c3-42ae-a139-e6f0098f23cd-plugins-config-volume\") pod \"1676214c-b5c3-42ae-a139-e6f0098f23cd\" (UID: \"1676214c-b5c3-42ae-a139-e6f0098f23cd\") " Nov 01 16:34:22.341815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.341692 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1676214c-b5c3-42ae-a139-e6f0098f23cd-kube-api-access-7xtkl\" (UniqueName: \"kubernetes.io/projected/1676214c-b5c3-42ae-a139-e6f0098f23cd-kube-api-access-7xtkl\") pod \"1676214c-b5c3-42ae-a139-e6f0098f23cd\" (UID: \"1676214c-b5c3-42ae-a139-e6f0098f23cd\") " Nov 01 16:34:22.342046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.342023 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1676214c-b5c3-42ae-a139-e6f0098f23cd-plugins-config-volume" pod "1676214c-b5c3-42ae-a139-e6f0098f23cd" (UID: "1676214c-b5c3-42ae-a139-e6f0098f23cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:22.342244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.342227 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1676214c-b5c3-42ae-a139-e6f0098f23cd-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.343967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.343937 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1676214c-b5c3-42ae-a139-e6f0098f23cd-kube-api-access-7xtkl" pod "1676214c-b5c3-42ae-a139-e6f0098f23cd" (UID: "1676214c-b5c3-42ae-a139-e6f0098f23cd"). InnerVolumeSpecName "kube-api-access-7xtkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:22.443818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.443788 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xtkl\" (UniqueName: \"kubernetes.io/projected/1676214c-b5c3-42ae-a139-e6f0098f23cd-kube-api-access-7xtkl\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.502585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.502555 4573 generic.go:356] "Generic (PLEG): container finished" podID="1676214c-b5c3-42ae-a139-e6f0098f23cd" containerID="5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23" exitCode=0 Nov 01 16:34:22.502740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.502617 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" Nov 01 16:34:22.502740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.502633 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" event={"ID":"1676214c-b5c3-42ae-a139-e6f0098f23cd","Type":"ContainerDied","Data":"5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23"} Nov 01 16:34:22.502740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.502685 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s" event={"ID":"1676214c-b5c3-42ae-a139-e6f0098f23cd","Type":"ContainerDied","Data":"c2085c781c3e3330a0015f26ae55f604233ad6af81d414b91404a8e7062d327e"} Nov 01 16:34:22.502740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.502708 4573 scope.go:122] "RemoveContainer" containerID="5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23" Nov 01 16:34:22.521833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.521805 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s"] Nov 01 16:34:22.525869 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.525841 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9f4dc9-7d9b75d459-qhb7s"] Nov 01 16:34:22.528190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.528162 4573 scope.go:122] "RemoveContainer" containerID="5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23" Nov 01 16:34:22.528517 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:22.528491 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23\": container with ID starting with 5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23 not found: ID does not exist" containerID="5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23" Nov 01 16:34:22.528592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.528530 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23"} err="failed to get container status \"5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23\": rpc error: code = NotFound desc = could not find container \"5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23\": container with ID starting with 5616347f59549678cc09b1a13876d3befca3ff480024d2ea559c2543a7102f23 not found: ID does not exist" Nov 01 16:34:22.662097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.662069 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 16:34:22.848400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848371 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-credential-socket\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.848593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848459 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-kube-api-access-ts7c9\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-kube-api-access-ts7c9\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.848593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848485 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-envoy\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.848593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848548 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-podinfo\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.848593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848584 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-token\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-token\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.848846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848609 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-socket\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.848846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848632 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90da294e-0e4c-4a46-920c-3f1df5e3343c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90da294e-0e4c-4a46-920c-3f1df5e3343c-istiod-ca-cert\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.848846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848659 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-data\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.848846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848703 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-credential-socket" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.848846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848718 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-certs\") pod \"90da294e-0e4c-4a46-920c-3f1df5e3343c\" (UID: \"90da294e-0e4c-4a46-920c-3f1df5e3343c\") " Nov 01 16:34:22.849158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848975 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-certs" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.849158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.848989 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-socket" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.849284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.849172 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-data" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.849284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.849256 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.849284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.849278 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.849463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.849295 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.849463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.849261 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90da294e-0e4c-4a46-920c-3f1df5e3343c-istiod-ca-cert" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:22.850785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.850763 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-kube-api-access-ts7c9" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "kube-api-access-ts7c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:22.850923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.850816 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-token" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:22.851166 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.851149 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-podinfo" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:34:22.851213 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.851161 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-envoy" pod "90da294e-0e4c-4a46-920c-3f1df5e3343c" (UID: "90da294e-0e4c-4a46-920c-3f1df5e3343c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:22.938387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.938336 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:22.949915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.949888 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.949915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.949912 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.950063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.949923 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.950063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.949932 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90da294e-0e4c-4a46-920c-3f1df5e3343c-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.950063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.949940 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90da294e-0e4c-4a46-920c-3f1df5e3343c-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:22.950063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:22.949948 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts7c9\" (UniqueName: \"kubernetes.io/projected/90da294e-0e4c-4a46-920c-3f1df5e3343c-kube-api-access-ts7c9\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:23.511827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.511788 4573 generic.go:356] "Generic (PLEG): container finished" podID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerID="b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae" exitCode=0 Nov 01 16:34:23.512406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.511842 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" Nov 01 16:34:23.512406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.511878 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" event={"ID":"90da294e-0e4c-4a46-920c-3f1df5e3343c","Type":"ContainerDied","Data":"b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae"} Nov 01 16:34:23.512406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.511944 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22" event={"ID":"90da294e-0e4c-4a46-920c-3f1df5e3343c","Type":"ContainerDied","Data":"6fcd927be0c5a1b0c57813534c30aa4f4b8af2aa3132e96fa78b2bb21071d068"} Nov 01 16:34:23.512406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.511970 4573 scope.go:122] "RemoveContainer" containerID="b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae" Nov 01 16:34:23.523225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.523204 4573 scope.go:122] "RemoveContainer" containerID="b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae" Nov 01 16:34:23.523501 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:23.523480 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae\": container with ID starting with b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae not found: ID does not exist" containerID="b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae" Nov 01 16:34:23.523551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.523512 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae"} err="failed to get container status \"b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae\": rpc error: code = NotFound desc = could not find container \"b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae\": container with ID starting with b47016720a228a5bdfd8d21c646051d23f90ca465cd9facbc4c9f0a9f9172bae not found: ID does not exist" Nov 01 16:34:23.535982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.535954 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22"] Nov 01 16:34:23.538621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.538599 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-dfff59-openshift-default-5fd958c56c-nfl22"] Nov 01 16:34:23.559054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:23.559031 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.131256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.131143 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.221403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.220552 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.301282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.301246 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.399645 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:24.399610 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:34:24.403895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.403863 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.458650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.458622 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1676214c-b5c3-42ae-a139-e6f0098f23cd" path="/var/lib/kubelet/pods/1676214c-b5c3-42ae-a139-e6f0098f23cd/volumes" Nov 01 16:34:24.459252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.459236 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" path="/var/lib/kubelet/pods/90da294e-0e4c-4a46-920c-3f1df5e3343c/volumes" Nov 01 16:34:24.469999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.469978 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:34:24.480339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480319 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-certs\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480392 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-kube-api-access-p475h\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-kube-api-access-p475h\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480479 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-envoy\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480504 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480501 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-socket\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480544 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-data\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480571 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-certs" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.480682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480591 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4247c96-f138-451a-9a73-305270e6a4a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4247c96-f138-451a-9a73-305270e6a4a0-istiod-ca-cert\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480638 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-istio-token\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-istio-token\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480671 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4247c96-f138-451a-9a73-305270e6a4a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4247c96-f138-451a-9a73-305270e6a4a0-istio-podinfo\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480723 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-credential-socket\") pod \"e4247c96-f138-451a-9a73-305270e6a4a0\" (UID: \"e4247c96-f138-451a-9a73-305270e6a4a0\") " Nov 01 16:34:24.480968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480745 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-socket" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.480968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.480791 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-data" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.481142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.481010 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-credential-socket" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.481142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.481051 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4247c96-f138-451a-9a73-305270e6a4a0-istiod-ca-cert" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:24.481401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.481309 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.481401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.481336 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.481401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.481352 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4247c96-f138-451a-9a73-305270e6a4a0-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.481401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.481367 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.481401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.481381 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.482657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.482625 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-envoy" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:34:24.482860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.482840 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-kube-api-access-p475h" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "kube-api-access-p475h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:24.482948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.482932 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-istio-token" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:24.482994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.482970 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4247c96-f138-451a-9a73-305270e6a4a0-istio-podinfo" pod "e4247c96-f138-451a-9a73-305270e6a4a0" (UID: "e4247c96-f138-451a-9a73-305270e6a4a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:34:24.518613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.518575 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.522832 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.522804 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4247c96-f138-451a-9a73-305270e6a4a0" containerID="30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a" exitCode=0 Nov 01 16:34:24.522952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.522859 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" Nov 01 16:34:24.522952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.522892 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" event={"ID":"e4247c96-f138-451a-9a73-305270e6a4a0","Type":"ContainerDied","Data":"30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a"} Nov 01 16:34:24.522952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.522949 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k" event={"ID":"e4247c96-f138-451a-9a73-305270e6a4a0","Type":"ContainerDied","Data":"d9abd883124758fd558cddc8d0710946b28c8bfdbd1f24b08a0d5d4887ae9b91"} Nov 01 16:34:24.523090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.522972 4573 scope.go:122] "RemoveContainer" containerID="30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a" Nov 01 16:34:24.534562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.534542 4573 scope.go:122] "RemoveContainer" containerID="30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a" Nov 01 16:34:24.534942 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:24.534920 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a\": container with ID starting with 30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a not found: ID does not exist" containerID="30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a" Nov 01 16:34:24.535017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.534953 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a"} err="failed to get container status \"30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a\": rpc error: code = NotFound desc = could not find container \"30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a\": container with ID starting with 30a4858179ed92c0ce8e2d6be2fe1e06352f92254cb55b818433f6117c60f37a not found: ID does not exist" Nov 01 16:34:24.547809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.547784 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k"] Nov 01 16:34:24.553847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.553821 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9f4dc9-openshift-default-66cbb8f7f5-8qf9k"] Nov 01 16:34:24.582947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.582921 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p475h\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-kube-api-access-p475h\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.582947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.582948 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4247c96-f138-451a-9a73-305270e6a4a0-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.583096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.582958 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4247c96-f138-451a-9a73-305270e6a4a0-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.583096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.582967 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4247c96-f138-451a-9a73-305270e6a4a0-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:24.614382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.614337 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.753253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.753175 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.873208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.873170 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:24.973161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:24.973107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:25.069800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:25.069697 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:25.398459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:25.398420 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:26.190502 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:26.190469 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:26.320492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:26.320439 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:26.457351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:26.457272 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" path="/var/lib/kubelet/pods/e4247c96-f138-451a-9a73-305270e6a4a0/volumes" Nov 01 16:34:27.146878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:27.146848 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:27.550135 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:27.550050 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:34:28.073039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:28.073011 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:28.620395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:28.620364 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:28.737343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:28.737313 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:28.839162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:28.839109 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:28.938575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:28.938493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:29.042535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:29.042493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:29.777971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:29.777931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:30.431846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:30.431811 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:30.678812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:30.678781 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:31.472244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:31.472211 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:31.574880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:31.574850 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:31.853209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:31.853113 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:32.303422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.303390 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294"] Nov 01 16:34:32.303969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.303952 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1676214c-b5c3-42ae-a139-e6f0098f23cd" containerName="payload-pre-processing" Nov 01 16:34:32.304032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.303971 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1676214c-b5c3-42ae-a139-e6f0098f23cd" containerName="payload-pre-processing" Nov 01 16:34:32.304032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.303982 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" Nov 01 16:34:32.304032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.303987 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" Nov 01 16:34:32.304032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304021 4573 cpu_manager.go:414] "containerMap: removing container" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" Nov 01 16:34:32.304032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304026 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" Nov 01 16:34:32.304032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304034 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8380742e-e617-4116-847a-2427d0841ae5" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304039 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8380742e-e617-4116-847a-2427d0841ae5" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304048 4573 cpu_manager.go:414] "containerMap: removing container" podUID="589709e8-9c48-49f5-91e4-d4717c7e3b42" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304053 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589709e8-9c48-49f5-91e4-d4717c7e3b42" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304058 4573 cpu_manager.go:414] "containerMap: removing container" podUID="dad7ef24-f0a2-4901-8301-fb6e72ccfb81" containerName="payload-pre-processing" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304063 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dad7ef24-f0a2-4901-8301-fb6e72ccfb81" containerName="payload-pre-processing" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304068 4573 cpu_manager.go:414] "containerMap: removing container" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304073 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304094 4573 cpu_manager.go:414] "containerMap: removing container" podUID="c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304099 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304112 4573 cpu_manager.go:414] "containerMap: removing container" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304116 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304183 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304191 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="839701ef-516c-4451-b096-50ca25cdcabc" containerName="main" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304199 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1676214c-b5c3-42ae-a139-e6f0098f23cd" containerName="payload-pre-processing" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304204 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4247c96-f138-451a-9a73-305270e6a4a0" containerName="istio-proxy" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304214 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="af872790-62d6-41e2-be43-07c7511b047b" containerName="main" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304220 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8380742e-e617-4116-847a-2427d0841ae5" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304225 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="589709e8-9c48-49f5-91e4-d4717c7e3b42" containerName="revoke-keys" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304232 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="dad7ef24-f0a2-4901-8301-fb6e72ccfb81" containerName="payload-pre-processing" Nov 01 16:34:32.304236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.304242 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="90da294e-0e4c-4a46-920c-3f1df5e3343c" containerName="istio-proxy" Nov 01 16:34:32.309001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.308984 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.313830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.313806 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294"] Nov 01 16:34:32.355902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.355875 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:34:32.355902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.355886 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:34:32.368408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.368381 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.368533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.368424 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.368533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.368516 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kft98\" (UniqueName: \"kubernetes.io/projected/31ec84e1-04a6-4699-85d7-cef04db16123-kube-api-access-kft98\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.436849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.436812 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:32.469097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.469059 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.469275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.469114 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.469275 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.469159 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kft98\" (UniqueName: \"kubernetes.io/projected/31ec84e1-04a6-4699-85d7-cef04db16123-kube-api-access-kft98\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.469694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.469670 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.471477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.471455 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.482307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.482271 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kft98\" (UniqueName: \"kubernetes.io/projected/31ec84e1-04a6-4699-85d7-cef04db16123-kube-api-access-kft98\") pod \"maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.537124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.537088 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:32.624372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.624344 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:32.914657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:32.914626 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294"] Nov 01 16:34:32.915483 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:34:32.915460 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ec84e1_04a6_4699_85d7_cef04db16123.slice/crio-2561176515ff46d700d9e27d64f8af79998fe250ce14ad25bb760f8d0f0876ac WatchSource:0}: Error finding container 2561176515ff46d700d9e27d64f8af79998fe250ce14ad25bb760f8d0f0876ac: Status 404 returned error can't find the container with id 2561176515ff46d700d9e27d64f8af79998fe250ce14ad25bb760f8d0f0876ac Nov 01 16:34:33.313025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:33.312991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:33.585514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:33.585415 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" event={"ID":"31ec84e1-04a6-4699-85d7-cef04db16123","Type":"ContainerStarted","Data":"4041c25e4d7cf5119c6771f1f38c97519289b1b98e2fda78dcc7dd27b975828d"} Nov 01 16:34:33.585514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:33.585475 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" event={"ID":"31ec84e1-04a6-4699-85d7-cef04db16123","Type":"ContainerStarted","Data":"2561176515ff46d700d9e27d64f8af79998fe250ce14ad25bb760f8d0f0876ac"} Nov 01 16:34:33.603793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:33.603733 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" podStartSLOduration=1.603719334 podStartE2EDuration="1.603719334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:34:33.600996846 +0000 UTC m=+2811.304162783" watchObservedRunningTime="2026-11-01 16:34:33.603719334 +0000 UTC m=+2811.306885315" Nov 01 16:34:34.046782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:34.046733 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:34.593759 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:34.593729 4573 generic.go:356] "Generic (PLEG): container finished" podID="31ec84e1-04a6-4699-85d7-cef04db16123" containerID="4041c25e4d7cf5119c6771f1f38c97519289b1b98e2fda78dcc7dd27b975828d" exitCode=0 Nov 01 16:34:34.594209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:34.593821 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" event={"ID":"31ec84e1-04a6-4699-85d7-cef04db16123","Type":"ContainerDied","Data":"4041c25e4d7cf5119c6771f1f38c97519289b1b98e2fda78dcc7dd27b975828d"} Nov 01 16:34:34.857071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:34.857037 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:35.556846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.556807 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:35.754097 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.754073 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:35.806343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.806312 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-openshift-ca\") pod \"31ec84e1-04a6-4699-85d7-cef04db16123\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " Nov 01 16:34:35.806518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.806433 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-xks-ca\") pod \"31ec84e1-04a6-4699-85d7-cef04db16123\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " Nov 01 16:34:35.806518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.806486 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ec84e1-04a6-4699-85d7-cef04db16123-kube-api-access-kft98\" (UniqueName: \"kubernetes.io/projected/31ec84e1-04a6-4699-85d7-cef04db16123-kube-api-access-kft98\") pod \"31ec84e1-04a6-4699-85d7-cef04db16123\" (UID: \"31ec84e1-04a6-4699-85d7-cef04db16123\") " Nov 01 16:34:35.806909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.806829 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-openshift-ca" pod "31ec84e1-04a6-4699-85d7-cef04db16123" (UID: "31ec84e1-04a6-4699-85d7-cef04db16123"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:35.807078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.807058 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:35.808847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.808823 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-xks-ca" pod "31ec84e1-04a6-4699-85d7-cef04db16123" (UID: "31ec84e1-04a6-4699-85d7-cef04db16123"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:35.808912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.808876 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ec84e1-04a6-4699-85d7-cef04db16123-kube-api-access-kft98" pod "31ec84e1-04a6-4699-85d7-cef04db16123" (UID: "31ec84e1-04a6-4699-85d7-cef04db16123"). InnerVolumeSpecName "kube-api-access-kft98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:35.908515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.908483 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ec84e1-04a6-4699-85d7-cef04db16123-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:35.908515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:35.908512 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kft98\" (UniqueName: \"kubernetes.io/projected/31ec84e1-04a6-4699-85d7-cef04db16123-kube-api-access-kft98\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:36.260261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:36.260226 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:36.441515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:36.441484 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:36.608721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:36.608699 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" Nov 01 16:34:36.608889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:36.608695 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294" event={"ID":"31ec84e1-04a6-4699-85d7-cef04db16123","Type":"ContainerDied","Data":"2561176515ff46d700d9e27d64f8af79998fe250ce14ad25bb760f8d0f0876ac"} Nov 01 16:34:36.608889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:36.608832 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2561176515ff46d700d9e27d64f8af79998fe250ce14ad25bb760f8d0f0876ac" Nov 01 16:34:37.063569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:37.063527 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:37.752616 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:37.752580 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:38.555295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:38.555265 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:39.172335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:39.172293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:39.902556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:39.902521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:40.672739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:40.672701 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:41.336744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:41.336710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:42.142524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:42.142485 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:42.754642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:42.754612 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:43.407668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:43.407611 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:44.077311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.077268 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:44.576116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.576083 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8"] Nov 01 16:34:44.576382 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.576339 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" podUID="2ff6f72e-575a-4019-9d99-2232e3638fdc" containerName="maas-api" containerID="cri-o://e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762" gracePeriod=30 Nov 01 16:34:44.619202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.619173 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:44.719225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.719186 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:44.823079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.823046 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:44.834928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.834868 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:34:44.869938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.869906 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294"] Nov 01 16:34:44.872505 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.872486 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f4a265-dt294"] Nov 01 16:34:44.893730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.893696 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:44.904849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.904821 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-maas-api-tls\") pod \"2ff6f72e-575a-4019-9d99-2232e3638fdc\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " Nov 01 16:34:44.904978 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.904951 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff6f72e-575a-4019-9d99-2232e3638fdc-kube-api-access-8mvkn\" (UniqueName: \"kubernetes.io/projected/2ff6f72e-575a-4019-9d99-2232e3638fdc-kube-api-access-8mvkn\") pod \"2ff6f72e-575a-4019-9d99-2232e3638fdc\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " Nov 01 16:34:44.905053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.905021 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff6f72e-575a-4019-9d99-2232e3638fdc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6f72e-575a-4019-9d99-2232e3638fdc-openshift-service-ca\") pod \"2ff6f72e-575a-4019-9d99-2232e3638fdc\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " Nov 01 16:34:44.905127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.905111 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-metrics-certs\") pod \"2ff6f72e-575a-4019-9d99-2232e3638fdc\" (UID: \"2ff6f72e-575a-4019-9d99-2232e3638fdc\") " Nov 01 16:34:44.905365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.905341 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff6f72e-575a-4019-9d99-2232e3638fdc-openshift-service-ca" pod "2ff6f72e-575a-4019-9d99-2232e3638fdc" (UID: "2ff6f72e-575a-4019-9d99-2232e3638fdc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:44.905624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.905592 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff6f72e-575a-4019-9d99-2232e3638fdc-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:44.907053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.907011 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-maas-api-tls" pod "2ff6f72e-575a-4019-9d99-2232e3638fdc" (UID: "2ff6f72e-575a-4019-9d99-2232e3638fdc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:44.907169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.907149 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-metrics-certs" pod "2ff6f72e-575a-4019-9d99-2232e3638fdc" (UID: "2ff6f72e-575a-4019-9d99-2232e3638fdc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:44.907343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.907324 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff6f72e-575a-4019-9d99-2232e3638fdc-kube-api-access-8mvkn" pod "2ff6f72e-575a-4019-9d99-2232e3638fdc" (UID: "2ff6f72e-575a-4019-9d99-2232e3638fdc"). InnerVolumeSpecName "kube-api-access-8mvkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:44.967359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:44.967326 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:45.006697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.006663 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:45.006697 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.006696 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ff6f72e-575a-4019-9d99-2232e3638fdc-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:45.007037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.006713 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mvkn\" (UniqueName: \"kubernetes.io/projected/2ff6f72e-575a-4019-9d99-2232e3638fdc-kube-api-access-8mvkn\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:45.325467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.325432 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:45.667023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.666993 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:45.675131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.675105 4573 generic.go:356] "Generic (PLEG): container finished" podID="2ff6f72e-575a-4019-9d99-2232e3638fdc" containerID="e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762" exitCode=0 Nov 01 16:34:45.675286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.675157 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" Nov 01 16:34:45.675286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.675191 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" event={"ID":"2ff6f72e-575a-4019-9d99-2232e3638fdc","Type":"ContainerDied","Data":"e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762"} Nov 01 16:34:45.675286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.675244 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8" event={"ID":"2ff6f72e-575a-4019-9d99-2232e3638fdc","Type":"ContainerDied","Data":"7375a932e2ec5e8c8ea56263a5ec9c2244826bc5c8100e73e962300bdefb2a95"} Nov 01 16:34:45.675286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.675266 4573 scope.go:122] "RemoveContainer" containerID="e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762" Nov 01 16:34:45.685715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.685697 4573 scope.go:122] "RemoveContainer" containerID="e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762" Nov 01 16:34:45.686002 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:45.685980 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762\": container with ID starting with e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762 not found: ID does not exist" containerID="e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762" Nov 01 16:34:45.686109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.686007 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762"} err="failed to get container status \"e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762\": rpc error: code = NotFound desc = could not find container \"e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762\": container with ID starting with e1c7c6701db7794669395fd5346f3ecafdcb19aafd90c32e54f204c762f21762 not found: ID does not exist" Nov 01 16:34:45.700023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.699994 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8"] Nov 01 16:34:45.702526 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:45.702507 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f4a265-68479df679-g6kw8"] Nov 01 16:34:46.453413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:46.453374 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:46.456909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:46.456888 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff6f72e-575a-4019-9d99-2232e3638fdc" path="/var/lib/kubelet/pods/2ff6f72e-575a-4019-9d99-2232e3638fdc/volumes" Nov 01 16:34:46.457317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:46.457296 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ec84e1-04a6-4699-85d7-cef04db16123" path="/var/lib/kubelet/pods/31ec84e1-04a6-4699-85d7-cef04db16123/volumes" Nov 01 16:34:47.257357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:47.257323 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:47.473550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:47.473520 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:34:48.064393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:48.064361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:48.749071 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:48.749035 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:49.559765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:49.559728 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:49.616471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:49.616431 4573 scope.go:122] "RemoveContainer" containerID="34b30a3782e647950f1635882abc5ea8375159cfada7f348c35fec62184a10ed" Nov 01 16:34:50.341662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:50.341635 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:50.961891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:50.961849 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:51.609799 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:51.609744 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:51.710780 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:51.710734 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:51.927072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:51.926992 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:52.415439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.415402 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559"] Nov 01 16:34:52.416206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.416175 4573 cpu_manager.go:414] "containerMap: removing container" podUID="31ec84e1-04a6-4699-85d7-cef04db16123" containerName="revoke-keys" Nov 01 16:34:52.416206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.416203 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ec84e1-04a6-4699-85d7-cef04db16123" containerName="revoke-keys" Nov 01 16:34:52.416423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.416223 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2ff6f72e-575a-4019-9d99-2232e3638fdc" containerName="maas-api" Nov 01 16:34:52.416423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.416231 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff6f72e-575a-4019-9d99-2232e3638fdc" containerName="maas-api" Nov 01 16:34:52.416423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.416348 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ec84e1-04a6-4699-85d7-cef04db16123" containerName="revoke-keys" Nov 01 16:34:52.416423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.416361 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff6f72e-575a-4019-9d99-2232e3638fdc" containerName="maas-api" Nov 01 16:34:52.421932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.421907 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.425251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.425214 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559"] Nov 01 16:34:52.502021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.501985 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:52.503630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.503609 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:34:52.503722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.503616 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:34:52.593050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.593018 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvtv7\" (UniqueName: \"kubernetes.io/projected/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-kube-api-access-jvtv7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.593241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.593074 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.593241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.593191 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.694079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.694002 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvtv7\" (UniqueName: \"kubernetes.io/projected/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-kube-api-access-jvtv7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.694079 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.694060 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.694489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.694127 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.694663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.694642 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.696456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.696428 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.701337 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.701311 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvtv7\" (UniqueName: \"kubernetes.io/projected/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-kube-api-access-jvtv7\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:52.737211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:52.737185 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:53.018031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:53.017994 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559"] Nov 01 16:34:53.019248 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:34:53.019215 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85e4f2d_cb85_4ea0_90d1_1bb39344e292.slice/crio-9e6e818e07f456972fa2cec72a9efc000077e802cc813cebc9ab5799975ee2e4 WatchSource:0}: Error finding container 9e6e818e07f456972fa2cec72a9efc000077e802cc813cebc9ab5799975ee2e4: Status 404 returned error can't find the container with id 9e6e818e07f456972fa2cec72a9efc000077e802cc813cebc9ab5799975ee2e4 Nov 01 16:34:53.376008 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:53.375979 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:53.728724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:53.728620 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" event={"ID":"d85e4f2d-cb85-4ea0-90d1-1bb39344e292","Type":"ContainerStarted","Data":"2a87ee4fbd7207fd9e2c88bd28c1bf7b8a9da19b0a233545b8b25029b602c61e"} Nov 01 16:34:53.728724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:53.728682 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" event={"ID":"d85e4f2d-cb85-4ea0-90d1-1bb39344e292","Type":"ContainerStarted","Data":"9e6e818e07f456972fa2cec72a9efc000077e802cc813cebc9ab5799975ee2e4"} Nov 01 16:34:53.750843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:53.744935 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" podStartSLOduration=1.744909958 podStartE2EDuration="1.744909958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:34:53.743768011 +0000 UTC m=+2831.446933938" watchObservedRunningTime="2026-11-01 16:34:53.744909958 +0000 UTC m=+2831.448075895" Nov 01 16:34:54.166965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:54.166933 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:54.363366 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:34:54.363328 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:34:54.740805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:54.740771 4573 generic.go:356] "Generic (PLEG): container finished" podID="d85e4f2d-cb85-4ea0-90d1-1bb39344e292" containerID="2a87ee4fbd7207fd9e2c88bd28c1bf7b8a9da19b0a233545b8b25029b602c61e" exitCode=0 Nov 01 16:34:54.741263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:54.740805 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" event={"ID":"d85e4f2d-cb85-4ea0-90d1-1bb39344e292","Type":"ContainerDied","Data":"2a87ee4fbd7207fd9e2c88bd28c1bf7b8a9da19b0a233545b8b25029b602c61e"} Nov 01 16:34:54.873538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:54.873506 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:55.630051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.629436 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:55.887034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.886980 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:55.928796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.928743 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-kube-api-access-jvtv7\" (UniqueName: \"kubernetes.io/projected/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-kube-api-access-jvtv7\") pod \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " Nov 01 16:34:55.928979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.928841 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-openshift-ca\") pod \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " Nov 01 16:34:55.928979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.928872 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-xks-ca\") pod \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\" (UID: \"d85e4f2d-cb85-4ea0-90d1-1bb39344e292\") " Nov 01 16:34:55.929230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.929206 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-openshift-ca" pod "d85e4f2d-cb85-4ea0-90d1-1bb39344e292" (UID: "d85e4f2d-cb85-4ea0-90d1-1bb39344e292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:34:55.929327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.929307 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:55.930931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.930900 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-kube-api-access-jvtv7" pod "d85e4f2d-cb85-4ea0-90d1-1bb39344e292" (UID: "d85e4f2d-cb85-4ea0-90d1-1bb39344e292"). InnerVolumeSpecName "kube-api-access-jvtv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:34:55.931042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:55.930943 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-xks-ca" pod "d85e4f2d-cb85-4ea0-90d1-1bb39344e292" (UID: "d85e4f2d-cb85-4ea0-90d1-1bb39344e292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:34:56.029871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:56.029830 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvtv7\" (UniqueName: \"kubernetes.io/projected/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-kube-api-access-jvtv7\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:56.029871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:56.029863 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d85e4f2d-cb85-4ea0-90d1-1bb39344e292-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:34:56.300454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:56.300371 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:56.756453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:56.756422 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" event={"ID":"d85e4f2d-cb85-4ea0-90d1-1bb39344e292","Type":"ContainerDied","Data":"9e6e818e07f456972fa2cec72a9efc000077e802cc813cebc9ab5799975ee2e4"} Nov 01 16:34:56.756453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:56.756467 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559" Nov 01 16:34:56.756715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:56.756471 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e6e818e07f456972fa2cec72a9efc000077e802cc813cebc9ab5799975ee2e4" Nov 01 16:34:56.958982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:56.958944 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:57.649508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:57.649471 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:58.266744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:58.266701 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:58.953793 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:58.953724 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:34:59.682414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:34:59.682382 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:00.358642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:00.358605 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:01.062850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:01.062820 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:01.748712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:01.748682 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:02.572365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:02.572334 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:02.649462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:02.649427 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:03.191218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:03.191180 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:03.422666 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:03.422634 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:03.431322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:03.431292 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559"] Nov 01 16:35:03.435288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:03.435265 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-383fde-b6559"] Nov 01 16:35:03.522811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:03.522691 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:03.916377 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:03.916350 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:04.274493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:04.274404 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:04.456779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:04.456734 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85e4f2d-cb85-4ea0-90d1-1bb39344e292" path="/var/lib/kubelet/pods/d85e4f2d-cb85-4ea0-90d1-1bb39344e292/volumes" Nov 01 16:35:04.984634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:04.984597 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:05.667531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:05.667501 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:05.852293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:05.852262 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:35:06.487741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:06.487710 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:07.174570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:07.174534 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:07.926324 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:07.926292 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:08.669857 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:08.669822 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:09.379153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:09.379121 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:10.102478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.102447 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:10.205704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.205670 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:10.303117 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.303090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:10.421553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.421517 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7"] Nov 01 16:35:10.422084 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.421737 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" containerID="cri-o://ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198" gracePeriod=30 Nov 01 16:35:10.452270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.452241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:10.868154 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.868116 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r"] Nov 01 16:35:10.868679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.868659 4573 cpu_manager.go:414] "containerMap: removing container" podUID="d85e4f2d-cb85-4ea0-90d1-1bb39344e292" containerName="revoke-keys" Nov 01 16:35:10.868805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.868681 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85e4f2d-cb85-4ea0-90d1-1bb39344e292" containerName="revoke-keys" Nov 01 16:35:10.868877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.868820 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85e4f2d-cb85-4ea0-90d1-1bb39344e292" containerName="revoke-keys" Nov 01 16:35:10.873473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.873449 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:10.879989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.879964 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r"] Nov 01 16:35:10.890312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.890284 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:10.911088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.911056 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:35:10.911227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.911175 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:35:10.976903 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.976868 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:10.977083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.976996 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rls2d\" (UniqueName: \"kubernetes.io/projected/363feabc-af44-47cb-bed4-5b3ae802cb09-kube-api-access-rls2d\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:10.977083 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:10.977052 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:11.077908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.077868 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:11.078118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.077957 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:11.078118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.078054 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rls2d\" (UniqueName: \"kubernetes.io/projected/363feabc-af44-47cb-bed4-5b3ae802cb09-kube-api-access-rls2d\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:11.078556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.078530 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:11.080380 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.080352 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:11.086252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.086219 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rls2d\" (UniqueName: \"kubernetes.io/projected/363feabc-af44-47cb-bed4-5b3ae802cb09-kube-api-access-rls2d\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:11.193134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.193046 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:11.468586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.468559 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r"] Nov 01 16:35:11.470732 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:35:11.470706 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363feabc_af44_47cb_bed4_5b3ae802cb09.slice/crio-98ddbeaec3b994eea70474cc67bcc208ab0dec53668611af257acafb383ae235 WatchSource:0}: Error finding container 98ddbeaec3b994eea70474cc67bcc208ab0dec53668611af257acafb383ae235: Status 404 returned error can't find the container with id 98ddbeaec3b994eea70474cc67bcc208ab0dec53668611af257acafb383ae235 Nov 01 16:35:11.770089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.770052 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:11.856723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.856690 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" event={"ID":"363feabc-af44-47cb-bed4-5b3ae802cb09","Type":"ContainerStarted","Data":"b2b223705116180c5c659de96adfe5d320dabf571b28d944aa6386d8dc43328a"} Nov 01 16:35:11.856896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.856799 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" event={"ID":"363feabc-af44-47cb-bed4-5b3ae802cb09","Type":"ContainerStarted","Data":"98ddbeaec3b994eea70474cc67bcc208ab0dec53668611af257acafb383ae235"} Nov 01 16:35:11.872703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:11.872657 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" podStartSLOduration=1.872642838 podStartE2EDuration="1.872642838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:35:11.870937643 +0000 UTC m=+2849.574103580" watchObservedRunningTime="2026-11-01 16:35:11.872642838 +0000 UTC m=+2849.575808774" Nov 01 16:35:12.464629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:12.464593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:12.864041 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:12.864007 4573 generic.go:356] "Generic (PLEG): container finished" podID="363feabc-af44-47cb-bed4-5b3ae802cb09" containerID="b2b223705116180c5c659de96adfe5d320dabf571b28d944aa6386d8dc43328a" exitCode=0 Nov 01 16:35:12.864471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:12.864093 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" event={"ID":"363feabc-af44-47cb-bed4-5b3ae802cb09","Type":"ContainerDied","Data":"b2b223705116180c5c659de96adfe5d320dabf571b28d944aa6386d8dc43328a"} Nov 01 16:35:13.155577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:13.155495 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:13.848419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:13.848387 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:14.013159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.013130 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:14.108954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.108924 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363feabc-af44-47cb-bed4-5b3ae802cb09-kube-api-access-rls2d\" (UniqueName: \"kubernetes.io/projected/363feabc-af44-47cb-bed4-5b3ae802cb09-kube-api-access-rls2d\") pod \"363feabc-af44-47cb-bed4-5b3ae802cb09\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " Nov 01 16:35:14.109139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.108998 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-openshift-ca\") pod \"363feabc-af44-47cb-bed4-5b3ae802cb09\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " Nov 01 16:35:14.109139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.109053 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-xks-ca\") pod \"363feabc-af44-47cb-bed4-5b3ae802cb09\" (UID: \"363feabc-af44-47cb-bed4-5b3ae802cb09\") " Nov 01 16:35:14.109426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.109403 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-openshift-ca" pod "363feabc-af44-47cb-bed4-5b3ae802cb09" (UID: "363feabc-af44-47cb-bed4-5b3ae802cb09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:14.111180 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.111153 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-xks-ca" pod "363feabc-af44-47cb-bed4-5b3ae802cb09" (UID: "363feabc-af44-47cb-bed4-5b3ae802cb09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:14.111288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.111219 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363feabc-af44-47cb-bed4-5b3ae802cb09-kube-api-access-rls2d" pod "363feabc-af44-47cb-bed4-5b3ae802cb09" (UID: "363feabc-af44-47cb-bed4-5b3ae802cb09"). InnerVolumeSpecName "kube-api-access-rls2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:14.211065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.211030 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:14.211065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.211062 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rls2d\" (UniqueName: \"kubernetes.io/projected/363feabc-af44-47cb-bed4-5b3ae802cb09-kube-api-access-rls2d\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:14.211065 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.211071 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/363feabc-af44-47cb-bed4-5b3ae802cb09-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:14.562651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.562585 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:14.878179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.878149 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" Nov 01 16:35:14.878365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.878156 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r" event={"ID":"363feabc-af44-47cb-bed4-5b3ae802cb09","Type":"ContainerDied","Data":"98ddbeaec3b994eea70474cc67bcc208ab0dec53668611af257acafb383ae235"} Nov 01 16:35:14.878365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:14.878280 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98ddbeaec3b994eea70474cc67bcc208ab0dec53668611af257acafb383ae235" Nov 01 16:35:15.387522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.387475 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:15.682807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.682781 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.725778 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-certs\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.725840 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-socket\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.725863 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-podinfo\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.725932 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-data\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.725961 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istiod-ca-cert\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.725999 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-credential-socket\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.726069 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-kube-api-access-hqt6n\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-kube-api-access-hqt6n\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.726109 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-token\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-token\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.726899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.726144 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-envoy\") pod \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\" (UID: \"e28d462e-43f1-4063-94a5-b4bedac2f2b5\") " Nov 01 16:35:15.727541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.727229 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-socket" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.727541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.727345 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-certs" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.727541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.727364 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-data" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.727710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.727607 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-credential-socket" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.727812 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.727788 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istiod-ca-cert" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:15.731339 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.731290 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-envoy" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:15.731708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.731682 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-kube-api-access-hqt6n" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "kube-api-access-hqt6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:15.731858 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.731787 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-token" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:15.732053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.732022 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-podinfo" pod "e28d462e-43f1-4063-94a5-b4bedac2f2b5" (UID: "e28d462e-43f1-4063-94a5-b4bedac2f2b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:15.827320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827287 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.827320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827315 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.827320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827326 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.827559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827335 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.827559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827344 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.827559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827353 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.827559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827362 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqt6n\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-kube-api-access-hqt6n\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.827559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827372 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.827559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.827380 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e28d462e-43f1-4063-94a5-b4bedac2f2b5-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:15.886902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.886867 4573 generic.go:356] "Generic (PLEG): container finished" podID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerID="ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198" exitCode=0 Nov 01 16:35:15.887076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.886917 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" Nov 01 16:35:15.887076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.886951 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" event={"ID":"e28d462e-43f1-4063-94a5-b4bedac2f2b5","Type":"ContainerDied","Data":"ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198"} Nov 01 16:35:15.887076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.887005 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7" event={"ID":"e28d462e-43f1-4063-94a5-b4bedac2f2b5","Type":"ContainerDied","Data":"1a44191c880e90d802fe901736f569590501164006265c1fa18fe529cfe7167b"} Nov 01 16:35:15.887076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.887029 4573 scope.go:122] "RemoveContainer" containerID="ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198" Nov 01 16:35:15.898113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.898098 4573 scope.go:122] "RemoveContainer" containerID="91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a" Nov 01 16:35:15.907849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.907825 4573 scope.go:122] "RemoveContainer" containerID="ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198" Nov 01 16:35:15.908135 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:35:15.908114 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198\": container with ID starting with ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198 not found: ID does not exist" containerID="ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198" Nov 01 16:35:15.908197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.908144 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198"} err="failed to get container status \"ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198\": rpc error: code = NotFound desc = could not find container \"ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198\": container with ID starting with ccb38c09d8477e5f80a1759f23c46a88eda7c6f73b74eb7b81cacd2d3c329198 not found: ID does not exist" Nov 01 16:35:15.908197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.908160 4573 scope.go:122] "RemoveContainer" containerID="91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a" Nov 01 16:35:15.908427 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:35:15.908408 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a\": container with ID starting with 91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a not found: ID does not exist" containerID="91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a" Nov 01 16:35:15.908494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.908432 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a"} err="failed to get container status \"91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a\": rpc error: code = NotFound desc = could not find container \"91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a\": container with ID starting with 91208aa3b1449fa40d8015f361055f4175b358cf6d16fcaf3166bcba9364ab2a not found: ID does not exist" Nov 01 16:35:15.911447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.911383 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7"] Nov 01 16:35:15.914359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:15.914339 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-383fde-openshift-default-5bf8ffbd59-9mgr7"] Nov 01 16:35:16.263836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:16.263739 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:16.456922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:16.456891 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" path="/var/lib/kubelet/pods/e28d462e-43f1-4063-94a5-b4bedac2f2b5/volumes" Nov 01 16:35:16.956411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:16.956376 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:17.662663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:17.662627 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:18.368864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:18.368834 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:19.061398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:19.061358 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:19.883805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:19.883743 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:20.559813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:20.559786 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:20.976973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:20.976938 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2"] Nov 01 16:35:21.042486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:21.042449 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:21.143463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:21.143414 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:21.265357 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:21.265276 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:21.959928 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:21.959894 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:22.490833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:22.490793 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:22.630591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:22.630557 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:22.753110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:22.753021 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:22.795916 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:22.795886 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r"] Nov 01 16:35:22.799031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:22.799005 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-a9a967-ngw5r"] Nov 01 16:35:22.856949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:22.856916 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:23.281291 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:23.281261 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:23.661501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:23.661460 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:24.363370 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:35:24.363334 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:35:24.368948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:24.368917 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:24.458289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:24.458254 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363feabc-af44-47cb-bed4-5b3ae802cb09" path="/var/lib/kubelet/pods/363feabc-af44-47cb-bed4-5b3ae802cb09/volumes" Nov 01 16:35:24.957391 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:24.957357 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:35:25.054436 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:25.054405 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:25.862550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:25.862513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:25.977850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:25.977812 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" podUID="e4e06231-d9cd-4796-a0a5-ef40859df1e5" containerName="payload-processing" containerID="cri-o://a5be05cfad5fdd5555e520ab3301dcb1b288a5327a1e0ed562b937041c127369" gracePeriod=25 Nov 01 16:35:26.565395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:26.565352 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:27.353076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:27.353037 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:28.149943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:28.149908 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:28.855531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:28.855494 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:29.451171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:29.451130 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:29.552332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:29.552300 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:29.651598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:29.651560 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:29.741001 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:29.740921 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2"] Nov 01 16:35:29.741283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:29.741255 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" containerID="cri-o://281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1" gracePeriod=30 Nov 01 16:35:29.749234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:29.749206 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr"] Nov 01 16:35:29.749479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:29.749452 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" containerID="cri-o://bfaa74f984a58884a2c3789f9708635fd6a1d0e6b3197dbea6b831c8e83ecb28" gracePeriod=30 Nov 01 16:35:29.784192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:29.784160 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:30.217072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.217041 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:30.369976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.369936 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq"] Nov 01 16:35:30.370417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.370403 4573 cpu_manager.go:414] "containerMap: removing container" podUID="363feabc-af44-47cb-bed4-5b3ae802cb09" containerName="revoke-keys" Nov 01 16:35:30.370477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.370419 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363feabc-af44-47cb-bed4-5b3ae802cb09" containerName="revoke-keys" Nov 01 16:35:30.370477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.370435 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" Nov 01 16:35:30.370477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.370440 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" Nov 01 16:35:30.370477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.370458 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" Nov 01 16:35:30.370477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.370463 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" Nov 01 16:35:30.370667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.370528 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" Nov 01 16:35:30.370667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.370541 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="363feabc-af44-47cb-bed4-5b3ae802cb09" containerName="revoke-keys" Nov 01 16:35:30.374068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.374052 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.378660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.378636 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:35:30.379652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.379626 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:35:30.383713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.383686 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq"] Nov 01 16:35:30.388269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.388244 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:30.484188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.484111 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.484188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.484160 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9bcv\" (UniqueName: \"kubernetes.io/projected/a4162778-8a53-4cf4-9f63-285e81489f9e-kube-api-access-m9bcv\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.484387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.484264 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.584888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.584857 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.585075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.584980 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.585075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.585025 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9bcv\" (UniqueName: \"kubernetes.io/projected/a4162778-8a53-4cf4-9f63-285e81489f9e-kube-api-access-m9bcv\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.585674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.585649 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.587123 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.587100 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.593440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.593417 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9bcv\" (UniqueName: \"kubernetes.io/projected/a4162778-8a53-4cf4-9f63-285e81489f9e-kube-api-access-m9bcv\") pod \"maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.689577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.689546 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:30.989986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:30.989934 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq"] Nov 01 16:35:30.992657 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:35:30.992622 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4162778_8a53_4cf4_9f63_285e81489f9e.slice/crio-8096c20d40368b9393a063f70ef5b4c7a096b56d49cd5d9c67180e763bc828b9 WatchSource:0}: Error finding container 8096c20d40368b9393a063f70ef5b4c7a096b56d49cd5d9c67180e763bc828b9: Status 404 returned error can't find the container with id 8096c20d40368b9393a063f70ef5b4c7a096b56d49cd5d9c67180e763bc828b9 Nov 01 16:35:31.255629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:31.255555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:32.002260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:32.002226 4573 generic.go:356] "Generic (PLEG): container finished" podID="a4162778-8a53-4cf4-9f63-285e81489f9e" containerID="2c4f0488d122127dc6ca2e821f6abc2ab40e014883199bdc3fa2313ac9d0aaec" exitCode=0 Nov 01 16:35:32.002458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:32.002315 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" event={"ID":"a4162778-8a53-4cf4-9f63-285e81489f9e","Type":"ContainerDied","Data":"2c4f0488d122127dc6ca2e821f6abc2ab40e014883199bdc3fa2313ac9d0aaec"} Nov 01 16:35:32.002458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:32.002379 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" event={"ID":"a4162778-8a53-4cf4-9f63-285e81489f9e","Type":"ContainerStarted","Data":"8096c20d40368b9393a063f70ef5b4c7a096b56d49cd5d9c67180e763bc828b9"} Nov 01 16:35:32.048796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:32.048699 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:32.540289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:32.540256 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:32.664901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:32.664873 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:33.151537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.151512 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:33.314092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.314005 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4162778-8a53-4cf4-9f63-285e81489f9e-kube-api-access-m9bcv\" (UniqueName: \"kubernetes.io/projected/a4162778-8a53-4cf4-9f63-285e81489f9e-kube-api-access-m9bcv\") pod \"a4162778-8a53-4cf4-9f63-285e81489f9e\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " Nov 01 16:35:33.314265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.314107 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-openshift-ca\") pod \"a4162778-8a53-4cf4-9f63-285e81489f9e\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " Nov 01 16:35:33.314265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.314140 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-xks-ca\") pod \"a4162778-8a53-4cf4-9f63-285e81489f9e\" (UID: \"a4162778-8a53-4cf4-9f63-285e81489f9e\") " Nov 01 16:35:33.314452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.314429 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-openshift-ca" pod "a4162778-8a53-4cf4-9f63-285e81489f9e" (UID: "a4162778-8a53-4cf4-9f63-285e81489f9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:33.314555 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.314536 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:33.316282 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.316255 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4162778-8a53-4cf4-9f63-285e81489f9e-kube-api-access-m9bcv" pod "a4162778-8a53-4cf4-9f63-285e81489f9e" (UID: "a4162778-8a53-4cf4-9f63-285e81489f9e"). InnerVolumeSpecName "kube-api-access-m9bcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:33.316422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.316306 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-xks-ca" pod "a4162778-8a53-4cf4-9f63-285e81489f9e" (UID: "a4162778-8a53-4cf4-9f63-285e81489f9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:33.372061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.372030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:33.416186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.416151 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4162778-8a53-4cf4-9f63-285e81489f9e-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:33.416186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:33.416188 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9bcv\" (UniqueName: \"kubernetes.io/projected/a4162778-8a53-4cf4-9f63-285e81489f9e-kube-api-access-m9bcv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:34.016699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:34.016669 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" Nov 01 16:35:34.017129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:34.016709 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq" event={"ID":"a4162778-8a53-4cf4-9f63-285e81489f9e","Type":"ContainerDied","Data":"8096c20d40368b9393a063f70ef5b4c7a096b56d49cd5d9c67180e763bc828b9"} Nov 01 16:35:34.017129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:34.016768 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8096c20d40368b9393a063f70ef5b4c7a096b56d49cd5d9c67180e763bc828b9" Nov 01 16:35:34.184305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:34.184277 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:34.515853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:34.515819 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" podUID="e4e06231-d9cd-4796-a0a5-ef40859df1e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:34.807488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:34.807457 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:35.002922 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.002897 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:35:35.040733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.040044 4573 generic.go:356] "Generic (PLEG): container finished" podID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerID="281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1" exitCode=0 Nov 01 16:35:35.040733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.040338 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" Nov 01 16:35:35.040733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.040445 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" event={"ID":"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00","Type":"ContainerDied","Data":"281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1"} Nov 01 16:35:35.040733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.040538 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2" event={"ID":"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00","Type":"ContainerDied","Data":"1767bfabc82297f6dff61de25716b43cd622bf5c750f3bb04fc6e70745e7176c"} Nov 01 16:35:35.040733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.040575 4573 scope.go:122] "RemoveContainer" containerID="281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1" Nov 01 16:35:35.047267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.044444 4573 generic.go:356] "Generic (PLEG): container finished" podID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerID="bfaa74f984a58884a2c3789f9708635fd6a1d0e6b3197dbea6b831c8e83ecb28" exitCode=0 Nov 01 16:35:35.047267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.044488 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" event={"ID":"54b84f57-6d13-4519-8167-50bafbbcedfe","Type":"ContainerDied","Data":"bfaa74f984a58884a2c3789f9708635fd6a1d0e6b3197dbea6b831c8e83ecb28"} Nov 01 16:35:35.047267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.044544 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" event={"ID":"54b84f57-6d13-4519-8167-50bafbbcedfe","Type":"ContainerDied","Data":"41eb3d29cc757dd43678d809000541589c0c32cf9956972b07195964b9a110db"} Nov 01 16:35:35.047267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.044568 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41eb3d29cc757dd43678d809000541589c0c32cf9956972b07195964b9a110db" Nov 01 16:35:35.058519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.058262 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:35:35.084194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.083113 4573 scope.go:122] "RemoveContainer" containerID="281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1" Nov 01 16:35:35.084194 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:35:35.083511 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1\": container with ID starting with 281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1 not found: ID does not exist" containerID="281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1" Nov 01 16:35:35.084194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.083539 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1"} err="failed to get container status \"281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1\": rpc error: code = NotFound desc = could not find container \"281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1\": container with ID starting with 281f04ece4cf3f3c7272807762eb5084ee3339fae81f8a4126912c9f22e30ae1 not found: ID does not exist" Nov 01 16:35:35.134904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.134877 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-certs\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.135034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.134912 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-podinfo\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.135034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.134935 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-podinfo\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.135034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.134967 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-data\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.135225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135126 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-data\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.135225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135152 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-certs" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.135225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135194 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-envoy\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.135405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135251 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-data" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.135405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135253 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-kube-api-access-856jk\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-kube-api-access-856jk\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.135405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135329 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-socket\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.135405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135388 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54b84f57-6d13-4519-8167-50bafbbcedfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b84f57-6d13-4519-8167-50bafbbcedfe-istiod-ca-cert\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.135662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135420 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-data" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.135662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135429 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-credential-socket\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.135662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135487 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-credential-socket\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.135662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135553 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-token\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-token\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.135662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135584 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-socket" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.135662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135589 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-token\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-token\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.135662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135639 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istiod-ca-cert\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.136175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135673 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-envoy\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.136175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135718 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-certs\") pod \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\" (UID: \"8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00\") " Nov 01 16:35:35.136175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.136147 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.136175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.136171 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.136410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.136186 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.136410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.136202 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.136410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135599 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-credential-socket" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.136410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.135905 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54b84f57-6d13-4519-8167-50bafbbcedfe-istiod-ca-cert" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:35.136410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.136102 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-credential-socket" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.136410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.136370 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istiod-ca-cert" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:35.136410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.136372 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-certs" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.137625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.137598 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-kube-api-access-856jk" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "kube-api-access-856jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.137742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.137659 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-podinfo" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:35.137742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.137674 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-podinfo" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:35:35.137901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.137789 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-envoy" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.137994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.137979 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-token" pod "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" (UID: "8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.138802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.138776 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-envoy" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.138909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.138784 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-token" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.237044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237007 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-socket\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.237205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237053 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-kube-api-access-z7n9s\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-kube-api-access-z7n9s\") pod \"54b84f57-6d13-4519-8167-50bafbbcedfe\" (UID: \"54b84f57-6d13-4519-8167-50bafbbcedfe\") " Nov 01 16:35:35.237306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237284 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-socket" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:35:35.237344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237338 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237350 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237360 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237369 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-856jk\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-kube-api-access-856jk\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237378 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54b84f57-6d13-4519-8167-50bafbbcedfe-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237387 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237396 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237404 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237411 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237420 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237429 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.237508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.237437 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.239144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.239121 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-kube-api-access-z7n9s" pod "54b84f57-6d13-4519-8167-50bafbbcedfe" (UID: "54b84f57-6d13-4519-8167-50bafbbcedfe"). InnerVolumeSpecName "kube-api-access-z7n9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:35.275853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.275774 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:35.338787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.338687 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54b84f57-6d13-4519-8167-50bafbbcedfe-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.338787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.338711 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7n9s\" (UniqueName: \"kubernetes.io/projected/54b84f57-6d13-4519-8167-50bafbbcedfe-kube-api-access-z7n9s\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:35.364709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.364679 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2"] Nov 01 16:35:35.367279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.367255 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-thzz2"] Nov 01 16:35:35.555489 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:35.555457 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:36.051481 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:36.051405 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr" Nov 01 16:35:36.073330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:36.073306 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr"] Nov 01 16:35:36.080344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:36.080319 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-a9a967-openshift-default-57f7644bd5-jfcdr"] Nov 01 16:35:36.245740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:36.245703 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:36.457288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:36.457255 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" path="/var/lib/kubelet/pods/54b84f57-6d13-4519-8167-50bafbbcedfe/volumes" Nov 01 16:35:36.457680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:36.457666 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" path="/var/lib/kubelet/pods/8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00/volumes" Nov 01 16:35:36.870425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:36.870395 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:37.547248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:37.547214 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:38.170024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:38.169991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:38.862258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:38.862230 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:39.747605 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:39.747569 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:40.366343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.366304 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:40.458453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.458423 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq"] Nov 01 16:35:40.521184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.521154 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:40.619856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.619779 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:40.912095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912003 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck"] Nov 01 16:35:40.912650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912627 4573 cpu_manager.go:414] "containerMap: removing container" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" Nov 01 16:35:40.912650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912648 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" Nov 01 16:35:40.912844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912664 4573 cpu_manager.go:414] "containerMap: removing container" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" Nov 01 16:35:40.912844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912669 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" Nov 01 16:35:40.912844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912687 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a4162778-8a53-4cf4-9f63-285e81489f9e" containerName="revoke-keys" Nov 01 16:35:40.912844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912692 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4162778-8a53-4cf4-9f63-285e81489f9e" containerName="revoke-keys" Nov 01 16:35:40.912844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912783 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e28d462e-43f1-4063-94a5-b4bedac2f2b5" containerName="istio-proxy" Nov 01 16:35:40.912844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912792 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4162778-8a53-4cf4-9f63-285e81489f9e" containerName="revoke-keys" Nov 01 16:35:40.912844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912804 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="54b84f57-6d13-4519-8167-50bafbbcedfe" containerName="istio-proxy" Nov 01 16:35:40.912844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.912810 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d2bc612-dcc3-4d1c-8aa6-24a0ba6d0b00" containerName="istio-proxy" Nov 01 16:35:40.917456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.917439 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:40.924923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.924897 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:35:40.925146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.924999 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:35:40.926501 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.926465 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck"] Nov 01 16:35:40.991856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.991829 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8gh\" (UniqueName: \"kubernetes.io/projected/2d04b34e-2bfd-4588-b268-6c98be4c260b-kube-api-access-7l8gh\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:40.991998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.991879 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:40.992054 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:40.991995 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:41.093318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:41.093290 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:41.093479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:41.093380 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8gh\" (UniqueName: \"kubernetes.io/projected/2d04b34e-2bfd-4588-b268-6c98be4c260b-kube-api-access-7l8gh\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:41.093479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:41.093409 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:41.094062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:41.094035 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:41.095426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:41.095399 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:41.101739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:41.101715 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8gh\" (UniqueName: \"kubernetes.io/projected/2d04b34e-2bfd-4588-b268-6c98be4c260b-kube-api-access-7l8gh\") pod \"maas-api-revoke-sub-472dee5d-gzbck\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:41.233943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:41.233872 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:41.507971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:41.507943 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck"] Nov 01 16:35:41.509200 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:35:41.509175 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d04b34e_2bfd_4588_b268_6c98be4c260b.slice/crio-30c6929af0a0a23a2c1bda375e0130b7fb06b9fbab61af74422a225095bbf705 WatchSource:0}: Error finding container 30c6929af0a0a23a2c1bda375e0130b7fb06b9fbab61af74422a225095bbf705: Status 404 returned error can't find the container with id 30c6929af0a0a23a2c1bda375e0130b7fb06b9fbab61af74422a225095bbf705 Nov 01 16:35:42.092114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:42.092033 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" event={"ID":"2d04b34e-2bfd-4588-b268-6c98be4c260b","Type":"ContainerStarted","Data":"7799e1e963c26726e0527f805900e537f70768e78bb4075ddce37147faa3363d"} Nov 01 16:35:42.092114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:42.092088 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" event={"ID":"2d04b34e-2bfd-4588-b268-6c98be4c260b","Type":"ContainerStarted","Data":"30c6929af0a0a23a2c1bda375e0130b7fb06b9fbab61af74422a225095bbf705"} Nov 01 16:35:42.108535 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:42.108490 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" podStartSLOduration=2.108477123 podStartE2EDuration="2.108477123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:35:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:35:42.10712845 +0000 UTC m=+2879.810294388" watchObservedRunningTime="2026-11-01 16:35:42.108477123 +0000 UTC m=+2879.811643059" Nov 01 16:35:43.099400 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:43.099365 4573 generic.go:356] "Generic (PLEG): container finished" podID="2d04b34e-2bfd-4588-b268-6c98be4c260b" containerID="7799e1e963c26726e0527f805900e537f70768e78bb4075ddce37147faa3363d" exitCode=0 Nov 01 16:35:43.099822 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:43.099403 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" event={"ID":"2d04b34e-2bfd-4588-b268-6c98be4c260b","Type":"ContainerDied","Data":"7799e1e963c26726e0527f805900e537f70768e78bb4075ddce37147faa3363d"} Nov 01 16:35:44.250372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.250342 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:44.328418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.328387 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d04b34e-2bfd-4588-b268-6c98be4c260b-kube-api-access-7l8gh\" (UniqueName: \"kubernetes.io/projected/2d04b34e-2bfd-4588-b268-6c98be4c260b-kube-api-access-7l8gh\") pod \"2d04b34e-2bfd-4588-b268-6c98be4c260b\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " Nov 01 16:35:44.328600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.328447 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-xks-ca\") pod \"2d04b34e-2bfd-4588-b268-6c98be4c260b\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " Nov 01 16:35:44.328600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.328538 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-openshift-ca\") pod \"2d04b34e-2bfd-4588-b268-6c98be4c260b\" (UID: \"2d04b34e-2bfd-4588-b268-6c98be4c260b\") " Nov 01 16:35:44.328958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.328934 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-openshift-ca" pod "2d04b34e-2bfd-4588-b268-6c98be4c260b" (UID: "2d04b34e-2bfd-4588-b268-6c98be4c260b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:44.330495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.330474 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d04b34e-2bfd-4588-b268-6c98be4c260b-kube-api-access-7l8gh" pod "2d04b34e-2bfd-4588-b268-6c98be4c260b" (UID: "2d04b34e-2bfd-4588-b268-6c98be4c260b"). InnerVolumeSpecName "kube-api-access-7l8gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:44.330582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.330549 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-xks-ca" pod "2d04b34e-2bfd-4588-b268-6c98be4c260b" (UID: "2d04b34e-2bfd-4588-b268-6c98be4c260b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:35:44.429310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.429281 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8gh\" (UniqueName: \"kubernetes.io/projected/2d04b34e-2bfd-4588-b268-6c98be4c260b-kube-api-access-7l8gh\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:44.429310 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.429309 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:44.429507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.429319 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d04b34e-2bfd-4588-b268-6c98be4c260b-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:44.515517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:44.515481 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" podUID="e4e06231-d9cd-4796-a0a5-ef40859df1e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:45.114299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:45.114263 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" Nov 01 16:35:45.114299 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:45.114287 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck" event={"ID":"2d04b34e-2bfd-4588-b268-6c98be4c260b","Type":"ContainerDied","Data":"30c6929af0a0a23a2c1bda375e0130b7fb06b9fbab61af74422a225095bbf705"} Nov 01 16:35:45.114532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:45.114359 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30c6929af0a0a23a2c1bda375e0130b7fb06b9fbab61af74422a225095bbf705" Nov 01 16:35:45.459352 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:45.459272 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" podUID="2e04b490-5013-4ff0-bd43-c529b49a559f" containerName="payload-processing" containerID="cri-o://85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510" gracePeriod=25 Nov 01 16:35:46.126130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.126100 4573 generic.go:356] "Generic (PLEG): container finished" podID="e4e06231-d9cd-4796-a0a5-ef40859df1e5" containerID="a5be05cfad5fdd5555e520ab3301dcb1b288a5327a1e0ed562b937041c127369" exitCode=0 Nov 01 16:35:46.126273 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.126146 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" event={"ID":"e4e06231-d9cd-4796-a0a5-ef40859df1e5","Type":"ContainerDied","Data":"a5be05cfad5fdd5555e520ab3301dcb1b288a5327a1e0ed562b937041c127369"} Nov 01 16:35:46.233837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.233776 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:35:46.348177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.348144 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e06231-d9cd-4796-a0a5-ef40859df1e5-kube-api-access-k2tsw\" (UniqueName: \"kubernetes.io/projected/e4e06231-d9cd-4796-a0a5-ef40859df1e5-kube-api-access-k2tsw\") pod \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\" (UID: \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\") " Nov 01 16:35:46.348356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.348267 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e06231-d9cd-4796-a0a5-ef40859df1e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e06231-d9cd-4796-a0a5-ef40859df1e5-plugins-config-volume\") pod \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\" (UID: \"e4e06231-d9cd-4796-a0a5-ef40859df1e5\") " Nov 01 16:35:46.348682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.348661 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e06231-d9cd-4796-a0a5-ef40859df1e5-plugins-config-volume" pod "e4e06231-d9cd-4796-a0a5-ef40859df1e5" (UID: "e4e06231-d9cd-4796-a0a5-ef40859df1e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:35:46.350241 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.350211 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e06231-d9cd-4796-a0a5-ef40859df1e5-kube-api-access-k2tsw" pod "e4e06231-d9cd-4796-a0a5-ef40859df1e5" (UID: "e4e06231-d9cd-4796-a0a5-ef40859df1e5"). InnerVolumeSpecName "kube-api-access-k2tsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:35:46.449775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.449713 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2tsw\" (UniqueName: \"kubernetes.io/projected/e4e06231-d9cd-4796-a0a5-ef40859df1e5-kube-api-access-k2tsw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:46.449775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:46.449740 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4e06231-d9cd-4796-a0a5-ef40859df1e5-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:35:47.134446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:47.134421 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" Nov 01 16:35:47.135114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:47.134462 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2" event={"ID":"e4e06231-d9cd-4796-a0a5-ef40859df1e5","Type":"ContainerDied","Data":"2c95887b1ed1dfc67be81c328fe3cfc6ad02707eb9fef168220364f7b7bb2d46"} Nov 01 16:35:47.135114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:47.134524 4573 scope.go:122] "RemoveContainer" containerID="a5be05cfad5fdd5555e520ab3301dcb1b288a5327a1e0ed562b937041c127369" Nov 01 16:35:47.152442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:47.152408 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2"] Nov 01 16:35:47.155686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:47.155664 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-a9a967-5dd77f664c-4v5m2"] Nov 01 16:35:48.456762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:48.456720 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e06231-d9cd-4796-a0a5-ef40859df1e5" path="/var/lib/kubelet/pods/e4e06231-d9cd-4796-a0a5-ef40859df1e5/volumes" Nov 01 16:35:49.701899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:49.701863 4573 scope.go:122] "RemoveContainer" containerID="543e2b19108e1d506e7c7029f91e381e65f86d15ed17e730cf67e2995bff750a" Nov 01 16:35:49.716157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:49.716136 4573 scope.go:122] "RemoveContainer" containerID="1e07f1bb08eab1c52ffc2b28882ccdbe4dbb37d1b025edbb2f3ad0ac059a7d50" Nov 01 16:35:49.724677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:49.724659 4573 scope.go:122] "RemoveContainer" containerID="6549f513835043a61f4f4878d192d04c9ac8ab0823537e2d808556986bf60d6a" Nov 01 16:35:49.733030 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:49.733015 4573 scope.go:122] "RemoveContainer" containerID="e5bd8c4e306b992fec497e8ab5bfc52053dffdd5aba1ac8468d0ef0b2676b1e8" Nov 01 16:35:49.741403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:49.741388 4573 scope.go:122] "RemoveContainer" containerID="bfaa74f984a58884a2c3789f9708635fd6a1d0e6b3197dbea6b831c8e83ecb28" Nov 01 16:35:51.323326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:51.323288 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" podUID="2e04b490-5013-4ff0-bd43-c529b49a559f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:35:52.971005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:52.970962 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:35:54.366902 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:35:54.366863 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:35:55.026409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:55.026372 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj"] Nov 01 16:35:55.028984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:55.028954 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14b4bf21-z7ldj"] Nov 01 16:35:55.989766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:55.989717 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:56.143078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:56.143041 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:56.168817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:56.168786 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq"] Nov 01 16:35:56.174387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:56.174362 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-164db6-52jdq"] Nov 01 16:35:56.231963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:56.231929 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:56.456492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:56.456459 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20cd89f5-e5e6-4f6b-968e-dccc8d393e53" path="/var/lib/kubelet/pods/20cd89f5-e5e6-4f6b-968e-dccc8d393e53/volumes" Nov 01 16:35:56.456878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:56.456863 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4162778-8a53-4cf4-9f63-285e81489f9e" path="/var/lib/kubelet/pods/a4162778-8a53-4cf4-9f63-285e81489f9e/volumes" Nov 01 16:35:56.668889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:56.668852 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:35:58.188365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:35:58.188333 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:36:01.323483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:01.323441 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" podUID="2e04b490-5013-4ff0-bd43-c529b49a559f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:36:02.727327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:02.727294 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:02.813242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:02.813210 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:03.032390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:03.032294 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j"] Nov 01 16:36:03.032975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:03.032940 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" containerID="cri-o://15b8aa6288d5971648ab6642e23e70ada85e1d13f797115d435730f013a51eaf" gracePeriod=30 Nov 01 16:36:03.060485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:03.060454 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:05.710861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:05.710834 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 16:36:05.838003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:05.837915 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e04b490-5013-4ff0-bd43-c529b49a559f-kube-api-access-js4f4\" (UniqueName: \"kubernetes.io/projected/2e04b490-5013-4ff0-bd43-c529b49a559f-kube-api-access-js4f4\") pod \"2e04b490-5013-4ff0-bd43-c529b49a559f\" (UID: \"2e04b490-5013-4ff0-bd43-c529b49a559f\") " Nov 01 16:36:05.838003 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:05.837963 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e04b490-5013-4ff0-bd43-c529b49a559f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e04b490-5013-4ff0-bd43-c529b49a559f-plugins-config-volume\") pod \"2e04b490-5013-4ff0-bd43-c529b49a559f\" (UID: \"2e04b490-5013-4ff0-bd43-c529b49a559f\") " Nov 01 16:36:05.838417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:05.838391 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e04b490-5013-4ff0-bd43-c529b49a559f-plugins-config-volume" pod "2e04b490-5013-4ff0-bd43-c529b49a559f" (UID: "2e04b490-5013-4ff0-bd43-c529b49a559f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:36:05.839932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:05.839914 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e04b490-5013-4ff0-bd43-c529b49a559f-kube-api-access-js4f4" pod "2e04b490-5013-4ff0-bd43-c529b49a559f" (UID: "2e04b490-5013-4ff0-bd43-c529b49a559f"). InnerVolumeSpecName "kube-api-access-js4f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:05.939125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:05.939093 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js4f4\" (UniqueName: \"kubernetes.io/projected/2e04b490-5013-4ff0-bd43-c529b49a559f-kube-api-access-js4f4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:05.939125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:05.939121 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e04b490-5013-4ff0-bd43-c529b49a559f-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:06.153435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.153404 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:36:06.253016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.252980 4573 generic.go:356] "Generic (PLEG): container finished" podID="2e04b490-5013-4ff0-bd43-c529b49a559f" containerID="85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510" exitCode=0 Nov 01 16:36:06.253179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.253021 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" Nov 01 16:36:06.253179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.253026 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" event={"ID":"2e04b490-5013-4ff0-bd43-c529b49a559f","Type":"ContainerDied","Data":"85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510"} Nov 01 16:36:06.253179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.253069 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq" event={"ID":"2e04b490-5013-4ff0-bd43-c529b49a559f","Type":"ContainerDied","Data":"762defcca9785600f7e55b4c1395c5209231e89b24fe1bd0d6bba9779c00876d"} Nov 01 16:36:06.253179 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.253095 4573 scope.go:122] "RemoveContainer" containerID="85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510" Nov 01 16:36:06.268149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.268125 4573 scope.go:122] "RemoveContainer" containerID="85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510" Nov 01 16:36:06.268450 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:36:06.268430 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510\": container with ID starting with 85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510 not found: ID does not exist" containerID="85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510" Nov 01 16:36:06.268532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.268458 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510"} err="failed to get container status \"85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510\": rpc error: code = NotFound desc = could not find container \"85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510\": container with ID starting with 85c453e0b64d186a0c60291bb1b6706de938e1be79b131a78bdcf30fd2d3f510 not found: ID does not exist" Nov 01 16:36:06.278689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.278664 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq"] Nov 01 16:36:06.285943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.285922 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-164db6-7848484d55-lzjsq"] Nov 01 16:36:06.459462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.459372 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e04b490-5013-4ff0-bd43-c529b49a559f" path="/var/lib/kubelet/pods/2e04b490-5013-4ff0-bd43-c529b49a559f/volumes" Nov 01 16:36:06.921144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.921105 4573 patch_prober.go:29] interesting pod/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 16:36:06.921662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:06.921161 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 16:36:08.272500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.272461 4573 generic.go:356] "Generic (PLEG): container finished" podID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerID="15b8aa6288d5971648ab6642e23e70ada85e1d13f797115d435730f013a51eaf" exitCode=0 Nov 01 16:36:08.273035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.272530 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" event={"ID":"aaf3d53a-d18e-48b9-ae20-21341d5f6fed","Type":"ContainerDied","Data":"15b8aa6288d5971648ab6642e23e70ada85e1d13f797115d435730f013a51eaf"} Nov 01 16:36:08.326883 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.326854 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:36:08.463130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463093 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-socket\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.463320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463162 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-token\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-token\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.463320 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463212 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-credential-socket\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.463459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463328 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-socket" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.463459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463384 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-envoy\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.463459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463435 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-data\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.463685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463476 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-certs\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.463685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463514 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-credential-socket" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.463685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463528 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istiod-ca-cert\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.463685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463568 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-podinfo\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.463685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463658 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-kube-api-access-9vlfw\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-kube-api-access-9vlfw\") pod \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\" (UID: \"aaf3d53a-d18e-48b9-ae20-21341d5f6fed\") " Nov 01 16:36:08.464028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463701 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-data" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.464028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.463795 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-certs" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.464028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.464000 4573 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-data\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.464028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.464021 4573 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.464252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.464038 4573 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-workload-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.464252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.464039 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istiod-ca-cert" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:36:08.464252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.464053 4573 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-credential-socket\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.465996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.465931 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-token" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:08.466425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.466402 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-podinfo" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 16:36:08.466532 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.466467 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-envoy" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:08.466613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.466594 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-kube-api-access-9vlfw" pod "aaf3d53a-d18e-48b9-ae20-21341d5f6fed" (UID: "aaf3d53a-d18e-48b9-ae20-21341d5f6fed"). InnerVolumeSpecName "kube-api-access-9vlfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:08.565099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.565012 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vlfw\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-kube-api-access-9vlfw\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.565099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.565042 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.565099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.565051 4573 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-envoy\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.565099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.565061 4573 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istiod-ca-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:08.565099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:08.565076 4573 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaf3d53a-d18e-48b9-ae20-21341d5f6fed-istio-podinfo\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:09.279977 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:09.279948 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" Nov 01 16:36:09.280475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:09.279977 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j" event={"ID":"aaf3d53a-d18e-48b9-ae20-21341d5f6fed","Type":"ContainerDied","Data":"8989d8536e46c64102256bbd846683daa455d6b288d9bad9e66eddb3b47dd7e0"} Nov 01 16:36:09.280475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:09.280043 4573 scope.go:122] "RemoveContainer" containerID="15b8aa6288d5971648ab6642e23e70ada85e1d13f797115d435730f013a51eaf" Nov 01 16:36:09.306456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:09.306422 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j"] Nov 01 16:36:09.313354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:09.313330 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-164db6-openshift-default-5c5fb4d6c7-r2t4j"] Nov 01 16:36:10.456885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:10.456845 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" path="/var/lib/kubelet/pods/aaf3d53a-d18e-48b9-ae20-21341d5f6fed/volumes" Nov 01 16:36:11.398550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:11.398517 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:11.498267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:11.498235 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:13.554175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:13.554145 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:15.765722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:15.765689 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m"] Nov 01 16:36:15.766240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:15.765925 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" podUID="a55ba933-be66-4aaa-a2a7-6b98b9e000e0" containerName="discovery" containerID="cri-o://46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4" gracePeriod=30 Nov 01 16:36:16.015541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.015518 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:36:16.028254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.028190 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-local-certs\") pod \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " Nov 01 16:36:16.028378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.028268 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-token\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-token\") pod \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " Nov 01 16:36:16.028378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.028316 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-ca-configmap\") pod \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " Nov 01 16:36:16.028378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.028344 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-dns-cert\") pod \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " Nov 01 16:36:16.028378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.028372 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-kubeconfig\") pod \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " Nov 01 16:36:16.028603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.028409 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-kube-api-access-fjlhf\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-kube-api-access-fjlhf\") pod \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " Nov 01 16:36:16.028824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.028781 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-ca-configmap" pod "a55ba933-be66-4aaa-a2a7-6b98b9e000e0" (UID: "a55ba933-be66-4aaa-a2a7-6b98b9e000e0"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:36:16.030421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.030394 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-kubeconfig" pod "a55ba933-be66-4aaa-a2a7-6b98b9e000e0" (UID: "a55ba933-be66-4aaa-a2a7-6b98b9e000e0"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:36:16.030512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.030450 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-token" pod "a55ba933-be66-4aaa-a2a7-6b98b9e000e0" (UID: "a55ba933-be66-4aaa-a2a7-6b98b9e000e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:16.030601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.030578 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-kube-api-access-fjlhf" pod "a55ba933-be66-4aaa-a2a7-6b98b9e000e0" (UID: "a55ba933-be66-4aaa-a2a7-6b98b9e000e0"). InnerVolumeSpecName "kube-api-access-fjlhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:16.030708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.030695 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-local-certs" pod "a55ba933-be66-4aaa-a2a7-6b98b9e000e0" (UID: "a55ba933-be66-4aaa-a2a7-6b98b9e000e0"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 16:36:16.030891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.030873 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-dns-cert" pod "a55ba933-be66-4aaa-a2a7-6b98b9e000e0" (UID: "a55ba933-be66-4aaa-a2a7-6b98b9e000e0"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:36:16.129152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.129118 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-cacerts\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-cacerts\") pod \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\" (UID: \"a55ba933-be66-4aaa-a2a7-6b98b9e000e0\") " Nov 01 16:36:16.129407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.129395 4573 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-local-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:16.129461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.129410 4573 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-token\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:16.129461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.129421 4573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-ca-configmap\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:16.129461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.129432 4573 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-csr-dns-cert\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:16.129461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.129443 4573 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-istio-kubeconfig\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:16.129461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.129454 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjlhf\" (UniqueName: \"kubernetes.io/projected/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-kube-api-access-fjlhf\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:16.131037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.131016 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-cacerts" pod "a55ba933-be66-4aaa-a2a7-6b98b9e000e0" (UID: "a55ba933-be66-4aaa-a2a7-6b98b9e000e0"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:36:16.230302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.230271 4573 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a55ba933-be66-4aaa-a2a7-6b98b9e000e0-cacerts\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:16.328244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.328164 4573 generic.go:356] "Generic (PLEG): container finished" podID="a55ba933-be66-4aaa-a2a7-6b98b9e000e0" containerID="46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4" exitCode=0 Nov 01 16:36:16.328244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.328210 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" Nov 01 16:36:16.328461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.328247 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" event={"ID":"a55ba933-be66-4aaa-a2a7-6b98b9e000e0","Type":"ContainerDied","Data":"46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4"} Nov 01 16:36:16.328461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.328301 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m" event={"ID":"a55ba933-be66-4aaa-a2a7-6b98b9e000e0","Type":"ContainerDied","Data":"68741f58b96281d03c7f98c1fb24c72d1d610766f998ca5bb7a380971094490c"} Nov 01 16:36:16.328461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.328323 4573 scope.go:122] "RemoveContainer" containerID="46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4" Nov 01 16:36:16.345277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.345258 4573 scope.go:122] "RemoveContainer" containerID="46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4" Nov 01 16:36:16.345537 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:36:16.345519 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4\": container with ID starting with 46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4 not found: ID does not exist" containerID="46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4" Nov 01 16:36:16.345603 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.345550 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4"} err="failed to get container status \"46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4\": rpc error: code = NotFound desc = could not find container \"46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4\": container with ID starting with 46f012c8ad41c47b75f5f555731c22712d32c40d6022611d5b806c56817796f4 not found: ID does not exist" Nov 01 16:36:16.353256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.353230 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m"] Nov 01 16:36:16.357570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.357547 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-27n7m"] Nov 01 16:36:16.456280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:16.456253 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55ba933-be66-4aaa-a2a7-6b98b9e000e0" path="/var/lib/kubelet/pods/a55ba933-be66-4aaa-a2a7-6b98b9e000e0/volumes" Nov 01 16:36:18.988919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:18.988883 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:24.363550 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:36:24.363471 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:36:27.239315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239283 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4"] Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239660 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e4e06231-d9cd-4796-a0a5-ef40859df1e5" containerName="payload-processing" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239671 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e06231-d9cd-4796-a0a5-ef40859df1e5" containerName="payload-processing" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239701 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a55ba933-be66-4aaa-a2a7-6b98b9e000e0" containerName="discovery" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239707 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55ba933-be66-4aaa-a2a7-6b98b9e000e0" containerName="discovery" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239718 4573 cpu_manager.go:414] "containerMap: removing container" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239723 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239731 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2e04b490-5013-4ff0-bd43-c529b49a559f" containerName="payload-processing" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239737 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e04b490-5013-4ff0-bd43-c529b49a559f" containerName="payload-processing" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239745 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2d04b34e-2bfd-4588-b268-6c98be4c260b" containerName="revoke-keys" Nov 01 16:36:27.239825 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239762 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d04b34e-2bfd-4588-b268-6c98be4c260b" containerName="revoke-keys" Nov 01 16:36:27.240173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239839 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55ba933-be66-4aaa-a2a7-6b98b9e000e0" containerName="discovery" Nov 01 16:36:27.240173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239850 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf3d53a-d18e-48b9-ae20-21341d5f6fed" containerName="istio-proxy" Nov 01 16:36:27.240173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239856 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e04b490-5013-4ff0-bd43-c529b49a559f" containerName="payload-processing" Nov 01 16:36:27.240173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239861 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d04b34e-2bfd-4588-b268-6c98be4c260b" containerName="revoke-keys" Nov 01 16:36:27.240173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.239869 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e06231-d9cd-4796-a0a5-ef40859df1e5" containerName="payload-processing" Nov 01 16:36:27.244188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.244172 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.248519 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.248495 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:36:27.250124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.250100 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:36:27.250448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.250431 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4"] Nov 01 16:36:27.321800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.321771 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.321920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.321884 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.321920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.321916 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vl8d\" (UniqueName: \"kubernetes.io/projected/6aa96599-7fce-4ee1-849a-0a212fdccc0d-kube-api-access-8vl8d\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.423089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.423053 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.423258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.423092 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vl8d\" (UniqueName: \"kubernetes.io/projected/6aa96599-7fce-4ee1-849a-0a212fdccc0d-kube-api-access-8vl8d\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.423258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.423127 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.423779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.423727 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.425452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.425433 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.446475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.446452 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vl8d\" (UniqueName: \"kubernetes.io/projected/6aa96599-7fce-4ee1-849a-0a212fdccc0d-kube-api-access-8vl8d\") pod \"maas-api-revoke-sub-85237e45-mhdf4\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.559542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.559465 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:27.837801 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:36:27.836868 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa96599_7fce_4ee1_849a_0a212fdccc0d.slice/crio-e4e7a33b8accba4f37038f59cc855c42f1c7191674ba05eea6e6ce8bb008ea9b WatchSource:0}: Error finding container e4e7a33b8accba4f37038f59cc855c42f1c7191674ba05eea6e6ce8bb008ea9b: Status 404 returned error can't find the container with id e4e7a33b8accba4f37038f59cc855c42f1c7191674ba05eea6e6ce8bb008ea9b Nov 01 16:36:27.839458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:27.839425 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4"] Nov 01 16:36:28.405782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:28.405740 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" event={"ID":"6aa96599-7fce-4ee1-849a-0a212fdccc0d","Type":"ContainerStarted","Data":"8c289f41892a8ff9d249e8fbf1941e54f54b91ceee15bf08c1f2ad99d6a8a1a8"} Nov 01 16:36:28.406195 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:28.405819 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" event={"ID":"6aa96599-7fce-4ee1-849a-0a212fdccc0d","Type":"ContainerStarted","Data":"e4e7a33b8accba4f37038f59cc855c42f1c7191674ba05eea6e6ce8bb008ea9b"} Nov 01 16:36:28.423442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:28.423347 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" podStartSLOduration=1.4233359239999999 podStartE2EDuration="1.423335924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:36:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:36:28.423011653 +0000 UTC m=+2926.126177589" watchObservedRunningTime="2026-11-01 16:36:28.423335924 +0000 UTC m=+2926.126501860" Nov 01 16:36:28.461656 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:28.461627 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:36:29.414184 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:29.414146 4573 generic.go:356] "Generic (PLEG): container finished" podID="6aa96599-7fce-4ee1-849a-0a212fdccc0d" containerID="8c289f41892a8ff9d249e8fbf1941e54f54b91ceee15bf08c1f2ad99d6a8a1a8" exitCode=0 Nov 01 16:36:29.414559 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:29.414237 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" event={"ID":"6aa96599-7fce-4ee1-849a-0a212fdccc0d","Type":"ContainerDied","Data":"8c289f41892a8ff9d249e8fbf1941e54f54b91ceee15bf08c1f2ad99d6a8a1a8"} Nov 01 16:36:30.557043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.557017 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:30.653316 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.653285 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-xks-ca\") pod \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " Nov 01 16:36:30.653483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.653325 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa96599-7fce-4ee1-849a-0a212fdccc0d-kube-api-access-8vl8d\" (UniqueName: \"kubernetes.io/projected/6aa96599-7fce-4ee1-849a-0a212fdccc0d-kube-api-access-8vl8d\") pod \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " Nov 01 16:36:30.653483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.653465 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-openshift-ca\") pod \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\" (UID: \"6aa96599-7fce-4ee1-849a-0a212fdccc0d\") " Nov 01 16:36:30.653879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.653847 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-openshift-ca" pod "6aa96599-7fce-4ee1-849a-0a212fdccc0d" (UID: "6aa96599-7fce-4ee1-849a-0a212fdccc0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:36:30.655309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.655284 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-xks-ca" pod "6aa96599-7fce-4ee1-849a-0a212fdccc0d" (UID: "6aa96599-7fce-4ee1-849a-0a212fdccc0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:36:30.655409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.655379 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa96599-7fce-4ee1-849a-0a212fdccc0d-kube-api-access-8vl8d" pod "6aa96599-7fce-4ee1-849a-0a212fdccc0d" (UID: "6aa96599-7fce-4ee1-849a-0a212fdccc0d"). InnerVolumeSpecName "kube-api-access-8vl8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:36:30.754193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.754116 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:30.754193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.754145 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6aa96599-7fce-4ee1-849a-0a212fdccc0d-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:30.754193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:30.754155 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vl8d\" (UniqueName: \"kubernetes.io/projected/6aa96599-7fce-4ee1-849a-0a212fdccc0d-kube-api-access-8vl8d\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:36:31.427802 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:31.427770 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" Nov 01 16:36:31.428026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:31.427797 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4" event={"ID":"6aa96599-7fce-4ee1-849a-0a212fdccc0d","Type":"ContainerDied","Data":"e4e7a33b8accba4f37038f59cc855c42f1c7191674ba05eea6e6ce8bb008ea9b"} Nov 01 16:36:31.428026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:31.427856 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4e7a33b8accba4f37038f59cc855c42f1c7191674ba05eea6e6ce8bb008ea9b" Nov 01 16:36:32.589577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:32.589544 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:38.731070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:38.731032 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:40.503859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:40.503824 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:36:49.806263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:49.806230 4573 scope.go:122] "RemoveContainer" containerID="9508774ca51006fa64b144e59de6936e5fccaf2b92775bce3ef9bf39724fd895" Nov 01 16:36:50.022551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:50.022515 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk"] Nov 01 16:36:50.024931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:50.024899 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27fd13bd-8vblk"] Nov 01 16:36:50.455619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:36:50.455584 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d70c578-0744-48fb-9851-012201fbbc20" path="/var/lib/kubelet/pods/6d70c578-0744-48fb-9851-012201fbbc20/volumes" Nov 01 16:36:54.362900 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:36:54.362864 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:37:00.021849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:00.021805 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d"] Nov 01 16:37:00.026473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:00.026446 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-66df19bb-ls59d"] Nov 01 16:37:00.195540 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:00.195503 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:37:00.457080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:00.457048 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7280418a-3eb2-48f4-813e-508c765be317" path="/var/lib/kubelet/pods/7280418a-3eb2-48f4-813e-508c765be317/volumes" Nov 01 16:37:11.023699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:11.023665 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw"] Nov 01 16:37:11.029325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:11.029305 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bff2f0d-4l2rw"] Nov 01 16:37:12.457645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:12.457612 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a3ab6bb-2808-417d-997e-6b3365904650" path="/var/lib/kubelet/pods/7a3ab6bb-2808-417d-997e-6b3365904650/volumes" Nov 01 16:37:20.023982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:20.023935 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg"] Nov 01 16:37:20.026294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:20.026028 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-061ca36f-hmgqg"] Nov 01 16:37:20.464218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:20.464176 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f57b93-a805-400f-b65f-f0e0140a3e31" path="/var/lib/kubelet/pods/f8f57b93-a805-400f-b65f-f0e0140a3e31/volumes" Nov 01 16:37:22.027537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:22.027501 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz"] Nov 01 16:37:22.032716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:22.032692 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3abcfa78-wh7dz"] Nov 01 16:37:22.457988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:22.457945 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="791d9dd2-b607-43af-a447-1650accc744f" path="/var/lib/kubelet/pods/791d9dd2-b607-43af-a447-1650accc744f/volumes" Nov 01 16:37:24.363666 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:37:24.363629 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:37:30.022912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:30.022874 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8"] Nov 01 16:37:30.026485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:30.026456 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecbb9205-55fp8"] Nov 01 16:37:30.457032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:30.456997 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b8ef3b4-e34c-45a9-afdc-55c2d2094f22" path="/var/lib/kubelet/pods/3b8ef3b4-e34c-45a9-afdc-55c2d2094f22/volumes" Nov 01 16:37:32.631497 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:32.631462 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:37:40.023118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:40.023086 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p"] Nov 01 16:37:40.025150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:40.025120 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7eca9fd4-zsp8p"] Nov 01 16:37:40.456584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:40.456553 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91f34694-5902-4ef4-bc9d-97ddb9e083f1" path="/var/lib/kubelet/pods/91f34694-5902-4ef4-bc9d-97ddb9e083f1/volumes" Nov 01 16:37:43.402565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:43.402534 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:37:43.414577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:43.414544 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:37:43.415100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:43.415082 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:37:43.426021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:43.425997 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:37:49.845579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:49.845543 4573 scope.go:122] "RemoveContainer" containerID="68a04ebbfb1497fec54c0c8b2492118df81cca56f00720bce64a4aee62a84d99" Nov 01 16:37:49.855301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:49.855277 4573 scope.go:122] "RemoveContainer" containerID="347d4bf6785d2d7f8341527f5320099cd46d7ae0147506de616b617b3cf4b80e" Nov 01 16:37:49.864186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:49.864164 4573 scope.go:122] "RemoveContainer" containerID="29c0812067cba1cf270891c0dbb03d22da7f14d74a61c7e638d55e31b98a7000" Nov 01 16:37:49.872444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:49.872420 4573 scope.go:122] "RemoveContainer" containerID="233c3d5571dd7d52a155fc2341818557f0f3ba35ee71a5263333e5425c985944" Nov 01 16:37:50.796366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:37:50.796330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:37:54.363232 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:37:54.363149 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:38:12.573369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:12.573330 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:38:13.390777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:13.390709 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:38:16.960759 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:16.960711 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz"] Nov 01 16:38:16.961177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:16.961114 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6aa96599-7fce-4ee1-849a-0a212fdccc0d" containerName="revoke-keys" Nov 01 16:38:16.961177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:16.961125 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa96599-7fce-4ee1-849a-0a212fdccc0d" containerName="revoke-keys" Nov 01 16:38:16.961253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:16.961205 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa96599-7fce-4ee1-849a-0a212fdccc0d" containerName="revoke-keys" Nov 01 16:38:16.964513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:16.964497 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:16.970972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:16.970943 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:38:16.971153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:16.971083 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:38:16.971946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:16.971921 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz"] Nov 01 16:38:17.070512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.070476 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr825\" (UniqueName: \"kubernetes.io/projected/bdddb1db-0469-4478-b288-badbc23d710f-kube-api-access-cr825\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.070512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.070512 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.070766 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.070654 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.171882 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.171848 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.172099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.172012 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr825\" (UniqueName: \"kubernetes.io/projected/bdddb1db-0469-4478-b288-badbc23d710f-kube-api-access-cr825\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.172099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.172046 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.172630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.172613 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.174435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.174407 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.179420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.179400 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr825\" (UniqueName: \"kubernetes.io/projected/bdddb1db-0469-4478-b288-badbc23d710f-kube-api-access-cr825\") pod \"maas-api-revoke-sub-c1e271a9-f64kz\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.280389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.280314 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:17.555677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:17.555649 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz"] Nov 01 16:38:17.558053 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:38:17.558021 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdddb1db_0469_4478_b288_badbc23d710f.slice/crio-eba80fd95dd5e02fc3aa6e66f3e22aa49cf95adb36ed5e5c62997783e173792b WatchSource:0}: Error finding container eba80fd95dd5e02fc3aa6e66f3e22aa49cf95adb36ed5e5c62997783e173792b: Status 404 returned error can't find the container with id eba80fd95dd5e02fc3aa6e66f3e22aa49cf95adb36ed5e5c62997783e173792b Nov 01 16:38:18.056915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:18.056876 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" event={"ID":"bdddb1db-0469-4478-b288-badbc23d710f","Type":"ContainerStarted","Data":"4c24b91f865f9f6533c69afd966b11fce2aa741a06c30e8292b33ce65a5ae135"} Nov 01 16:38:18.057378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:18.056935 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" event={"ID":"bdddb1db-0469-4478-b288-badbc23d710f","Type":"ContainerStarted","Data":"eba80fd95dd5e02fc3aa6e66f3e22aa49cf95adb36ed5e5c62997783e173792b"} Nov 01 16:38:18.073569 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:18.073516 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" podStartSLOduration=2.073501104 podStartE2EDuration="2.073501104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:38:18.072131345 +0000 UTC m=+3035.775297281" watchObservedRunningTime="2026-11-01 16:38:18.073501104 +0000 UTC m=+3035.776667039" Nov 01 16:38:19.064880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:19.064847 4573 generic.go:356] "Generic (PLEG): container finished" podID="bdddb1db-0469-4478-b288-badbc23d710f" containerID="4c24b91f865f9f6533c69afd966b11fce2aa741a06c30e8292b33ce65a5ae135" exitCode=0 Nov 01 16:38:19.065321 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:19.064934 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" event={"ID":"bdddb1db-0469-4478-b288-badbc23d710f","Type":"ContainerDied","Data":"4c24b91f865f9f6533c69afd966b11fce2aa741a06c30e8292b33ce65a5ae135"} Nov 01 16:38:20.023670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.023630 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk"] Nov 01 16:38:20.029148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.029092 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2"] Nov 01 16:38:20.031947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.031918 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a11d637-m75zk"] Nov 01 16:38:20.033735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.033714 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd98059-8x8b2"] Nov 01 16:38:20.206568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.206541 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:20.302000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.301919 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdddb1db-0469-4478-b288-badbc23d710f-kube-api-access-cr825\" (UniqueName: \"kubernetes.io/projected/bdddb1db-0469-4478-b288-badbc23d710f-kube-api-access-cr825\") pod \"bdddb1db-0469-4478-b288-badbc23d710f\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " Nov 01 16:38:20.302157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.302010 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-xks-ca\") pod \"bdddb1db-0469-4478-b288-badbc23d710f\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " Nov 01 16:38:20.302157 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.302041 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-openshift-ca\") pod \"bdddb1db-0469-4478-b288-badbc23d710f\" (UID: \"bdddb1db-0469-4478-b288-badbc23d710f\") " Nov 01 16:38:20.302458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.302423 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-openshift-ca" pod "bdddb1db-0469-4478-b288-badbc23d710f" (UID: "bdddb1db-0469-4478-b288-badbc23d710f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:38:20.304235 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.304210 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-xks-ca" pod "bdddb1db-0469-4478-b288-badbc23d710f" (UID: "bdddb1db-0469-4478-b288-badbc23d710f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:38:20.304364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.304323 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdddb1db-0469-4478-b288-badbc23d710f-kube-api-access-cr825" pod "bdddb1db-0469-4478-b288-badbc23d710f" (UID: "bdddb1db-0469-4478-b288-badbc23d710f"). InnerVolumeSpecName "kube-api-access-cr825". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:38:20.402701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.402666 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:20.402701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.402697 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bdddb1db-0469-4478-b288-badbc23d710f-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:20.402701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.402708 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr825\" (UniqueName: \"kubernetes.io/projected/bdddb1db-0469-4478-b288-badbc23d710f-kube-api-access-cr825\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:20.456085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.456056 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b0dc0f7-debd-4c72-a795-47b2f17144d5" path="/var/lib/kubelet/pods/6b0dc0f7-debd-4c72-a795-47b2f17144d5/volumes" Nov 01 16:38:20.456530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:20.456515 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb64a0d-5f25-47f4-8397-83fc89da9381" path="/var/lib/kubelet/pods/bcb64a0d-5f25-47f4-8397-83fc89da9381/volumes" Nov 01 16:38:21.078797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:21.078770 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" Nov 01 16:38:21.078997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:21.078760 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz" event={"ID":"bdddb1db-0469-4478-b288-badbc23d710f","Type":"ContainerDied","Data":"eba80fd95dd5e02fc3aa6e66f3e22aa49cf95adb36ed5e5c62997783e173792b"} Nov 01 16:38:21.078997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:21.078898 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eba80fd95dd5e02fc3aa6e66f3e22aa49cf95adb36ed5e5c62997783e173792b" Nov 01 16:38:24.363615 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:38:24.363579 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:38:24.821522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:24.821440 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:38:24.922439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:24.922405 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:38:28.036035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:28.035998 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:38:31.393945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:31.393905 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:38:32.681289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:32.681253 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:38:35.310683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.310644 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk"] Nov 01 16:38:35.311092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.311041 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bdddb1db-0469-4478-b288-badbc23d710f" containerName="revoke-keys" Nov 01 16:38:35.311092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.311052 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdddb1db-0469-4478-b288-badbc23d710f" containerName="revoke-keys" Nov 01 16:38:35.311186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.311137 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdddb1db-0469-4478-b288-badbc23d710f" containerName="revoke-keys" Nov 01 16:38:35.315970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.315949 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.328962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.328933 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk"] Nov 01 16:38:35.351347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.351319 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx"] Nov 01 16:38:35.355342 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.355317 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.371459 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.371431 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx"] Nov 01 16:38:35.446699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.446665 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94f6q\" (UniqueName: \"kubernetes.io/projected/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-kube-api-access-94f6q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.446931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.446706 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.446931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.446727 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.446931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.446864 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.446931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.446929 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.447202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.446981 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.447202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447023 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.447202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447067 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.447202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447112 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.447202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447150 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxrr\" (UniqueName: \"kubernetes.io/projected/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-kube-api-access-zdxrr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.447415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447227 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.447415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447255 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.447415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447288 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.447415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447303 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.447415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447361 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.447415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447410 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.447627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447465 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.447627 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.447484 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.548782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.548729 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94f6q\" (UniqueName: \"kubernetes.io/projected/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-kube-api-access-94f6q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.548958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.548810 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.548958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.548840 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.548958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.548864 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.548958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.548889 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.548958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.548920 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.548958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.548951 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.548983 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549012 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549038 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxrr\" (UniqueName: \"kubernetes.io/projected/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-kube-api-access-zdxrr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549104 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549132 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549184 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549210 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549238 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549264 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549269 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549287 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.549326 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549313 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.550011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549363 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.550011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549460 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.550011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549631 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.550011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549676 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.550011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549701 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.550011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549699 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.550011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.549728 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.550372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.550032 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.550372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.550329 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.551623 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.551600 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.551998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.551978 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.552077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.552004 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.552077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.552050 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.557101 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.557081 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94f6q\" (UniqueName: \"kubernetes.io/projected/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-kube-api-access-94f6q\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.557205 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.557186 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.557512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.557490 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc68abf9-50c7-4707-bd62-64fe92c4d9f9-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-r9wbx\" (UID: \"fc68abf9-50c7-4707-bd62-64fe92c4d9f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.557631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.557615 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxrr\" (UniqueName: \"kubernetes.io/projected/c1e3f0a9-1576-4410-b17e-be1afb4b73a8-kube-api-access-zdxrr\") pod \"maas-default-gateway-openshift-default-7f5667f86c-gnmrk\" (UID: \"c1e3f0a9-1576-4410-b17e-be1afb4b73a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.631533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.631505 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:35.674080 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.674045 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:35.709874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.708484 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl"] Nov 01 16:38:35.719909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.714654 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.719909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.718475 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl"] Nov 01 16:38:35.719909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.719458 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:38:35.719909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.719528 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:38:35.751737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.751704 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.752107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.752084 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjzwg\" (UniqueName: \"kubernetes.io/projected/e647bfc3-44ac-4aaa-84ee-7e45701194b2-kube-api-access-sjzwg\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.752278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.752259 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.853671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.853636 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.853860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.853740 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjzwg\" (UniqueName: \"kubernetes.io/projected/e647bfc3-44ac-4aaa-84ee-7e45701194b2-kube-api-access-sjzwg\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.853860 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.853816 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.854306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.854287 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.856037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.856019 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.861507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.861488 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjzwg\" (UniqueName: \"kubernetes.io/projected/e647bfc3-44ac-4aaa-84ee-7e45701194b2-kube-api-access-sjzwg\") pod \"maas-api-revoke-sub-2e1f5b5f-9dkvl\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:35.911904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.911875 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk"] Nov 01 16:38:35.913491 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:38:35.913461 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e3f0a9_1576_4410_b17e_be1afb4b73a8.slice/crio-818a4fdb6b664e7ca51dbbddf3c568da9aa3533ab61c7e2d9fe9aa0c814c99a2 WatchSource:0}: Error finding container 818a4fdb6b664e7ca51dbbddf3c568da9aa3533ab61c7e2d9fe9aa0c814c99a2: Status 404 returned error can't find the container with id 818a4fdb6b664e7ca51dbbddf3c568da9aa3533ab61c7e2d9fe9aa0c814c99a2 Nov 01 16:38:35.915583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.915557 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.915647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.915618 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.915689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.915648 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.977063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.977034 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx"] Nov 01 16:38:35.978655 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:38:35.978624 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc68abf9_50c7_4707_bd62_64fe92c4d9f9.slice/crio-8a882df28561a98ef83b5505872c9cf58e1c5b216e54e8d9228bf08fb1cc154a WatchSource:0}: Error finding container 8a882df28561a98ef83b5505872c9cf58e1c5b216e54e8d9228bf08fb1cc154a: Status 404 returned error can't find the container with id 8a882df28561a98ef83b5505872c9cf58e1c5b216e54e8d9228bf08fb1cc154a Nov 01 16:38:35.981268 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.981236 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.981348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.981299 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:35.981390 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:35.981350 4573 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Nov 01 16:38:36.036172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.036150 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:36.176901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.176802 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" event={"ID":"c1e3f0a9-1576-4410-b17e-be1afb4b73a8","Type":"ContainerStarted","Data":"92df8242dafed37329c18a00dbdd7cc7e3ec27103e69d477bb3ab3ee0da978bd"} Nov 01 16:38:36.176901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.176869 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" event={"ID":"c1e3f0a9-1576-4410-b17e-be1afb4b73a8","Type":"ContainerStarted","Data":"818a4fdb6b664e7ca51dbbddf3c568da9aa3533ab61c7e2d9fe9aa0c814c99a2"} Nov 01 16:38:36.178910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.178877 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" event={"ID":"fc68abf9-50c7-4707-bd62-64fe92c4d9f9","Type":"ContainerStarted","Data":"855458da9b91d74e3ff308315f95d0cc2d3c9ff3bb6e47fcbe86205cc8a9716a"} Nov 01 16:38:36.179067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.178933 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" event={"ID":"fc68abf9-50c7-4707-bd62-64fe92c4d9f9","Type":"ContainerStarted","Data":"8a882df28561a98ef83b5505872c9cf58e1c5b216e54e8d9228bf08fb1cc154a"} Nov 01 16:38:36.196940 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.196891 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" podStartSLOduration=1.196875114 podStartE2EDuration="1.196875114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:38:36.194654564 +0000 UTC m=+3053.897820500" watchObservedRunningTime="2026-11-01 16:38:36.196875114 +0000 UTC m=+3053.900041050" Nov 01 16:38:36.216066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.216009 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" podStartSLOduration=1.21599105 podStartE2EDuration="1.21599105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:38:36.211680494 +0000 UTC m=+3053.914846435" watchObservedRunningTime="2026-11-01 16:38:36.21599105 +0000 UTC m=+3053.919156986" Nov 01 16:38:36.332874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.332719 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl"] Nov 01 16:38:36.337408 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:38:36.337372 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode647bfc3_44ac_4aaa_84ee_7e45701194b2.slice/crio-04587a1143b312ae03d816e01e6a9877c599372bdafc4d2de8cd99102bfe6028 WatchSource:0}: Error finding container 04587a1143b312ae03d816e01e6a9877c599372bdafc4d2de8cd99102bfe6028: Status 404 returned error can't find the container with id 04587a1143b312ae03d816e01e6a9877c599372bdafc4d2de8cd99102bfe6028 Nov 01 16:38:36.631927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.631891 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:36.633372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.633348 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gnmrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Nov 01 16:38:36.633493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.633407 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" podUID="c1e3f0a9-1576-4410-b17e-be1afb4b73a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Nov 01 16:38:36.686774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:36.675845 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:37.225289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:37.224761 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" event={"ID":"e647bfc3-44ac-4aaa-84ee-7e45701194b2","Type":"ContainerStarted","Data":"6531bb78e3c62617c43c3dbe29a8495a310e51fbdd204ab28cd90a27e408f538"} Nov 01 16:38:37.225289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:37.224820 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" event={"ID":"e647bfc3-44ac-4aaa-84ee-7e45701194b2","Type":"ContainerStarted","Data":"04587a1143b312ae03d816e01e6a9877c599372bdafc4d2de8cd99102bfe6028"} Nov 01 16:38:37.241932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:37.241876 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" podStartSLOduration=2.241862672 podStartE2EDuration="2.241862672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:38:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:38:37.238472812 +0000 UTC m=+3054.941638754" watchObservedRunningTime="2026-11-01 16:38:37.241862672 +0000 UTC m=+3054.945028619" Nov 01 16:38:37.632957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:37.632922 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gnmrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Nov 01 16:38:37.633346 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:37.632982 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" podUID="c1e3f0a9-1576-4410-b17e-be1afb4b73a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Nov 01 16:38:37.679703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:37.679352 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r9wbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Nov 01 16:38:37.679703 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:37.679412 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" podUID="fc68abf9-50c7-4707-bd62-64fe92c4d9f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Nov 01 16:38:38.235309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:38.235268 4573 generic.go:356] "Generic (PLEG): container finished" podID="e647bfc3-44ac-4aaa-84ee-7e45701194b2" containerID="6531bb78e3c62617c43c3dbe29a8495a310e51fbdd204ab28cd90a27e408f538" exitCode=0 Nov 01 16:38:38.235957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:38.235843 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" event={"ID":"e647bfc3-44ac-4aaa-84ee-7e45701194b2","Type":"ContainerDied","Data":"6531bb78e3c62617c43c3dbe29a8495a310e51fbdd204ab28cd90a27e408f538"} Nov 01 16:38:38.632163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:38.632131 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gnmrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Nov 01 16:38:38.632422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:38.632396 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" podUID="c1e3f0a9-1576-4410-b17e-be1afb4b73a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Nov 01 16:38:38.682770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:38.682570 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r9wbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Nov 01 16:38:38.682770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:38.682643 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" podUID="fc68abf9-50c7-4707-bd62-64fe92c4d9f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Nov 01 16:38:39.519981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.519898 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:39.632022 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.631987 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gnmrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Nov 01 16:38:39.632206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.632043 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" podUID="c1e3f0a9-1576-4410-b17e-be1afb4b73a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Nov 01 16:38:39.633293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.633273 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-openshift-ca\") pod \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " Nov 01 16:38:39.633861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.633408 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e647bfc3-44ac-4aaa-84ee-7e45701194b2-kube-api-access-sjzwg\" (UniqueName: \"kubernetes.io/projected/e647bfc3-44ac-4aaa-84ee-7e45701194b2-kube-api-access-sjzwg\") pod \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " Nov 01 16:38:39.633861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.633465 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-xks-ca\") pod \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\" (UID: \"e647bfc3-44ac-4aaa-84ee-7e45701194b2\") " Nov 01 16:38:39.633861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.633664 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-openshift-ca" pod "e647bfc3-44ac-4aaa-84ee-7e45701194b2" (UID: "e647bfc3-44ac-4aaa-84ee-7e45701194b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:38:39.633861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.633831 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:39.635577 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.635554 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-xks-ca" pod "e647bfc3-44ac-4aaa-84ee-7e45701194b2" (UID: "e647bfc3-44ac-4aaa-84ee-7e45701194b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:38:39.635668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.635622 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e647bfc3-44ac-4aaa-84ee-7e45701194b2-kube-api-access-sjzwg" pod "e647bfc3-44ac-4aaa-84ee-7e45701194b2" (UID: "e647bfc3-44ac-4aaa-84ee-7e45701194b2"). InnerVolumeSpecName "kube-api-access-sjzwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:38:39.675177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.675144 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r9wbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Nov 01 16:38:39.678771 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.675419 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" podUID="fc68abf9-50c7-4707-bd62-64fe92c4d9f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Nov 01 16:38:39.734567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.734507 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjzwg\" (UniqueName: \"kubernetes.io/projected/e647bfc3-44ac-4aaa-84ee-7e45701194b2-kube-api-access-sjzwg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:39.734567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:39.734540 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e647bfc3-44ac-4aaa-84ee-7e45701194b2-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:40.248679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:40.248651 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" Nov 01 16:38:40.248679 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:40.248675 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl" event={"ID":"e647bfc3-44ac-4aaa-84ee-7e45701194b2","Type":"ContainerDied","Data":"04587a1143b312ae03d816e01e6a9877c599372bdafc4d2de8cd99102bfe6028"} Nov 01 16:38:40.248937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:40.248718 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04587a1143b312ae03d816e01e6a9877c599372bdafc4d2de8cd99102bfe6028" Nov 01 16:38:40.632100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:40.632061 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gnmrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Nov 01 16:38:40.632293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:40.632122 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" podUID="c1e3f0a9-1576-4410-b17e-be1afb4b73a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Nov 01 16:38:40.678783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:40.678244 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r9wbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Nov 01 16:38:40.678783 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:40.678320 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" podUID="fc68abf9-50c7-4707-bd62-64fe92c4d9f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Nov 01 16:38:41.632286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:41.632249 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gnmrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Nov 01 16:38:41.632743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:41.632303 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" podUID="c1e3f0a9-1576-4410-b17e-be1afb4b73a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Nov 01 16:38:41.679776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:41.678164 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r9wbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Nov 01 16:38:41.679776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:41.678224 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" podUID="fc68abf9-50c7-4707-bd62-64fe92c4d9f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Nov 01 16:38:42.631981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:42.631937 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-gnmrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" start-of-body= Nov 01 16:38:42.632178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:42.632013 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" podUID="c1e3f0a9-1576-4410-b17e-be1afb4b73a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.75:15021/healthz/ready\": dial tcp 10.133.1.75:15021: connect: connection refused" Nov 01 16:38:42.675311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:42.675272 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r9wbx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" start-of-body= Nov 01 16:38:42.675733 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:42.675327 4573 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" podUID="fc68abf9-50c7-4707-bd62-64fe92c4d9f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": dial tcp 10.133.1.76:15021: connect: connection refused" Nov 01 16:38:44.205529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:44.205317 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:44.206137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:44.205941 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:44.207028 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:44.207005 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-gnmrk" Nov 01 16:38:44.347593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:44.347557 4573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:44.348350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:44.348205 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:44.348979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:44.348948 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" Nov 01 16:38:46.648848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.648798 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:38:46.719264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.719225 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr"] Nov 01 16:38:46.719721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.719703 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e647bfc3-44ac-4aaa-84ee-7e45701194b2" containerName="revoke-keys" Nov 01 16:38:46.719845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.719723 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e647bfc3-44ac-4aaa-84ee-7e45701194b2" containerName="revoke-keys" Nov 01 16:38:46.719921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.719842 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e647bfc3-44ac-4aaa-84ee-7e45701194b2" containerName="revoke-keys" Nov 01 16:38:46.723270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.723248 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.727122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.727097 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:38:46.728266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.728246 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:38:46.730433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.730410 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr"] Nov 01 16:38:46.809189 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.809143 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgnwv\" (UniqueName: \"kubernetes.io/projected/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-kube-api-access-bgnwv\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.809363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.809223 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.809413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.809382 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.910231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.910150 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.910231 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.910210 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgnwv\" (UniqueName: \"kubernetes.io/projected/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-kube-api-access-bgnwv\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.910420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.910249 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.910887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.910858 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.912544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.912527 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:46.918973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:46.918952 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgnwv\" (UniqueName: \"kubernetes.io/projected/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-kube-api-access-bgnwv\") pod \"maas-api-revoke-sub-ec843018-658zr\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:47.040474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:47.040441 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:47.327265 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:47.327224 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr"] Nov 01 16:38:47.329785 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:38:47.329240 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c157e8a_6c9e_4aa9_a81c_7015cac4801a.slice/crio-3712c27cabd5a2aaddcc07fe9c2e82af24c7dc0b38ad62528dddfc62645cdadf WatchSource:0}: Error finding container 3712c27cabd5a2aaddcc07fe9c2e82af24c7dc0b38ad62528dddfc62645cdadf: Status 404 returned error can't find the container with id 3712c27cabd5a2aaddcc07fe9c2e82af24c7dc0b38ad62528dddfc62645cdadf Nov 01 16:38:48.310932 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:48.310894 4573 generic.go:356] "Generic (PLEG): container finished" podID="1c157e8a-6c9e-4aa9-a81c-7015cac4801a" containerID="f0d9c63dcc3126253b753d197145a4952a9d70048f1cf54b914c32979b35fc16" exitCode=0 Nov 01 16:38:48.311433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:48.311030 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" event={"ID":"1c157e8a-6c9e-4aa9-a81c-7015cac4801a","Type":"ContainerDied","Data":"f0d9c63dcc3126253b753d197145a4952a9d70048f1cf54b914c32979b35fc16"} Nov 01 16:38:48.311433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:48.311081 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" event={"ID":"1c157e8a-6c9e-4aa9-a81c-7015cac4801a","Type":"ContainerStarted","Data":"3712c27cabd5a2aaddcc07fe9c2e82af24c7dc0b38ad62528dddfc62645cdadf"} Nov 01 16:38:49.556234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.556198 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:49.743485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.743452 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-openshift-ca\") pod \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " Nov 01 16:38:49.743723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.743556 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-xks-ca\") pod \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " Nov 01 16:38:49.743723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.743657 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-kube-api-access-bgnwv\" (UniqueName: \"kubernetes.io/projected/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-kube-api-access-bgnwv\") pod \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\" (UID: \"1c157e8a-6c9e-4aa9-a81c-7015cac4801a\") " Nov 01 16:38:49.743961 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.743933 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-openshift-ca" pod "1c157e8a-6c9e-4aa9-a81c-7015cac4801a" (UID: "1c157e8a-6c9e-4aa9-a81c-7015cac4801a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:38:49.746017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.745987 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-kube-api-access-bgnwv" pod "1c157e8a-6c9e-4aa9-a81c-7015cac4801a" (UID: "1c157e8a-6c9e-4aa9-a81c-7015cac4801a"). InnerVolumeSpecName "kube-api-access-bgnwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:38:49.746224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.746198 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-xks-ca" pod "1c157e8a-6c9e-4aa9-a81c-7015cac4801a" (UID: "1c157e8a-6c9e-4aa9-a81c-7015cac4801a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:38:49.845673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.845628 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:49.845673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.845673 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgnwv\" (UniqueName: \"kubernetes.io/projected/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-kube-api-access-bgnwv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:49.845900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.845691 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c157e8a-6c9e-4aa9-a81c-7015cac4801a-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:38:49.909267 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.909228 4573 scope.go:122] "RemoveContainer" containerID="4e60b8c6241e00eb0c74e20b390df9085d41641c78ac8716fa9f9b58eb89255a" Nov 01 16:38:49.921347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.921316 4573 scope.go:122] "RemoveContainer" containerID="f76f116203928cde5027a28e72e1435fbe5c3e0c2507a4a577a8ce3d29b93e56" Nov 01 16:38:49.932094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.932071 4573 scope.go:122] "RemoveContainer" containerID="461aaf765d7418c3e77c5c41c6182f1d29456a86a973966f9a196f34d75f9797" Nov 01 16:38:49.942807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.942738 4573 scope.go:122] "RemoveContainer" containerID="10c8de49972c51f9f2815deae9d2c25c5bf45a303aae6e0b26f9a76cb952a1ca" Nov 01 16:38:49.952817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.952793 4573 scope.go:122] "RemoveContainer" containerID="c01b89ea08955af7aaaf9cd3e512e53a052106319ff78f014306ff5b7326c356" Nov 01 16:38:49.963201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.963176 4573 scope.go:122] "RemoveContainer" containerID="a470048b500f723a50a8a48e74d78a8fa97dace43d1f27c4021904076931c02e" Nov 01 16:38:49.973311 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.973288 4573 scope.go:122] "RemoveContainer" containerID="fb011cb7cdc383a24b61292b0b534a776c94801de0e68e5d6ddbb8b107eefaec" Nov 01 16:38:49.983614 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:49.983590 4573 scope.go:122] "RemoveContainer" containerID="6eb08577ed4ba136fb6cf2d4fc5951501ee63a1c16dd5961d8e546d3922c2d39" Nov 01 16:38:50.331581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:50.331496 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" event={"ID":"1c157e8a-6c9e-4aa9-a81c-7015cac4801a","Type":"ContainerDied","Data":"3712c27cabd5a2aaddcc07fe9c2e82af24c7dc0b38ad62528dddfc62645cdadf"} Nov 01 16:38:50.331581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:50.331557 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3712c27cabd5a2aaddcc07fe9c2e82af24c7dc0b38ad62528dddfc62645cdadf" Nov 01 16:38:50.331792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:50.331628 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr" Nov 01 16:38:54.364060 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:38:54.364028 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:38:59.339590 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:38:59.339552 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:39:02.025774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:02.025295 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx"] Nov 01 16:39:02.032243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:02.032194 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28a7d65-czrvx"] Nov 01 16:39:02.457698 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:02.457663 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8380742e-e617-4116-847a-2427d0841ae5" path="/var/lib/kubelet/pods/8380742e-e617-4116-847a-2427d0841ae5/volumes" Nov 01 16:39:03.029188 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:03.029156 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h"] Nov 01 16:39:03.031169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:03.031141 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j"] Nov 01 16:39:03.037225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:03.037199 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ff31524-xhf4j"] Nov 01 16:39:03.039122 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:03.039103 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e942cb9b-q7z8h"] Nov 01 16:39:04.456402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:04.456368 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589709e8-9c48-49f5-91e4-d4717c7e3b42" path="/var/lib/kubelet/pods/589709e8-9c48-49f5-91e4-d4717c7e3b42/volumes" Nov 01 16:39:04.456892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:04.456874 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4" path="/var/lib/kubelet/pods/c3e04db5-df2a-4b58-b21d-68c2d6bc0ef4/volumes" Nov 01 16:39:13.116338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.116304 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g"] Nov 01 16:39:13.116746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.116733 4573 cpu_manager.go:414] "containerMap: removing container" podUID="1c157e8a-6c9e-4aa9-a81c-7015cac4801a" containerName="revoke-keys" Nov 01 16:39:13.116746 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.116762 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c157e8a-6c9e-4aa9-a81c-7015cac4801a" containerName="revoke-keys" Nov 01 16:39:13.116889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.116879 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c157e8a-6c9e-4aa9-a81c-7015cac4801a" containerName="revoke-keys" Nov 01 16:39:13.119971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.119955 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.125019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.124997 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:39:13.125233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.125216 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:39:13.127138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.127112 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g"] Nov 01 16:39:13.146404 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.146378 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.146520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.146428 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-969kb\" (UniqueName: \"kubernetes.io/projected/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-kube-api-access-969kb\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.146520 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.146456 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.247234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.247201 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.247394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.247293 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.247394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.247332 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-969kb\" (UniqueName: \"kubernetes.io/projected/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-kube-api-access-969kb\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.247976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.247953 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.249494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.249471 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.255201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.255182 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-969kb\" (UniqueName: \"kubernetes.io/projected/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-kube-api-access-969kb\") pod \"maas-api-revoke-sub-046e1343-9n67g\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.436111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.436028 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:13.717787 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:13.717063 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g"] Nov 01 16:39:13.726973 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:39:13.726935 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91faa95d_32b4_42a4_8ef9_b5d35762f4f4.slice/crio-da56b8ea22ad3131c8cd9190118e706e85b41718cd14c90118147c380192331c WatchSource:0}: Error finding container da56b8ea22ad3131c8cd9190118e706e85b41718cd14c90118147c380192331c: Status 404 returned error can't find the container with id da56b8ea22ad3131c8cd9190118e706e85b41718cd14c90118147c380192331c Nov 01 16:39:14.492150 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:14.492110 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" event={"ID":"91faa95d-32b4-42a4-8ef9-b5d35762f4f4","Type":"ContainerStarted","Data":"86de13504acef3f35fe9786d23fa7cf24106d972afabadb7d9e36f666c1a925b"} Nov 01 16:39:14.492608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:14.492171 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" event={"ID":"91faa95d-32b4-42a4-8ef9-b5d35762f4f4","Type":"ContainerStarted","Data":"da56b8ea22ad3131c8cd9190118e706e85b41718cd14c90118147c380192331c"} Nov 01 16:39:14.508997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:14.508950 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" podStartSLOduration=1.508935513 podStartE2EDuration="1.508935513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:39:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:39:14.507957153 +0000 UTC m=+3092.211123090" watchObservedRunningTime="2026-11-01 16:39:14.508935513 +0000 UTC m=+3092.212101446" Nov 01 16:39:15.498936 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:15.498906 4573 generic.go:356] "Generic (PLEG): container finished" podID="91faa95d-32b4-42a4-8ef9-b5d35762f4f4" containerID="86de13504acef3f35fe9786d23fa7cf24106d972afabadb7d9e36f666c1a925b" exitCode=0 Nov 01 16:39:15.499412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:15.498948 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" event={"ID":"91faa95d-32b4-42a4-8ef9-b5d35762f4f4","Type":"ContainerDied","Data":"86de13504acef3f35fe9786d23fa7cf24106d972afabadb7d9e36f666c1a925b"} Nov 01 16:39:16.639690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.639666 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:16.681144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.681118 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-kube-api-access-969kb\" (UniqueName: \"kubernetes.io/projected/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-kube-api-access-969kb\") pod \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " Nov 01 16:39:16.681314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.681174 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-xks-ca\") pod \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " Nov 01 16:39:16.681394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.681329 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-openshift-ca\") pod \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\" (UID: \"91faa95d-32b4-42a4-8ef9-b5d35762f4f4\") " Nov 01 16:39:16.681693 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.681668 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-openshift-ca" pod "91faa95d-32b4-42a4-8ef9-b5d35762f4f4" (UID: "91faa95d-32b4-42a4-8ef9-b5d35762f4f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:39:16.683197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.683166 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-xks-ca" pod "91faa95d-32b4-42a4-8ef9-b5d35762f4f4" (UID: "91faa95d-32b4-42a4-8ef9-b5d35762f4f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:39:16.683309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.683285 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-kube-api-access-969kb" pod "91faa95d-32b4-42a4-8ef9-b5d35762f4f4" (UID: "91faa95d-32b4-42a4-8ef9-b5d35762f4f4"). InnerVolumeSpecName "kube-api-access-969kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:39:16.782088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.782022 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:16.782088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.782047 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:16.782088 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:16.782058 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-969kb\" (UniqueName: \"kubernetes.io/projected/91faa95d-32b4-42a4-8ef9-b5d35762f4f4-kube-api-access-969kb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:17.510419 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:17.510389 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" Nov 01 16:39:17.510649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:17.510421 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g" event={"ID":"91faa95d-32b4-42a4-8ef9-b5d35762f4f4","Type":"ContainerDied","Data":"da56b8ea22ad3131c8cd9190118e706e85b41718cd14c90118147c380192331c"} Nov 01 16:39:17.510649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:17.510475 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da56b8ea22ad3131c8cd9190118e706e85b41718cd14c90118147c380192331c" Nov 01 16:39:23.316521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.316484 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm"] Nov 01 16:39:23.316941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.316927 4573 cpu_manager.go:414] "containerMap: removing container" podUID="91faa95d-32b4-42a4-8ef9-b5d35762f4f4" containerName="revoke-keys" Nov 01 16:39:23.316987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.316942 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91faa95d-32b4-42a4-8ef9-b5d35762f4f4" containerName="revoke-keys" Nov 01 16:39:23.317032 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.317004 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="91faa95d-32b4-42a4-8ef9-b5d35762f4f4" containerName="revoke-keys" Nov 01 16:39:23.320294 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.320267 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.324478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.324456 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:39:23.325408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.325388 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:39:23.328468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.328417 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm"] Nov 01 16:39:23.445896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.445863 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.446072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.445928 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.446072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.446040 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmcrb\" (UniqueName: \"kubernetes.io/projected/6e6beffd-779a-4c53-a78d-a5beda2ba056-kube-api-access-fmcrb\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.546995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.546962 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmcrb\" (UniqueName: \"kubernetes.io/projected/6e6beffd-779a-4c53-a78d-a5beda2ba056-kube-api-access-fmcrb\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.547255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.547031 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.547255 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.547098 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.547847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.547827 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.549503 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.549485 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.555367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.555346 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmcrb\" (UniqueName: \"kubernetes.io/projected/6e6beffd-779a-4c53-a78d-a5beda2ba056-kube-api-access-fmcrb\") pod \"maas-api-revoke-sub-d3fba056-rs4qm\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.637181 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.637151 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:23.887281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.887201 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:39:23.913861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:23.913836 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm"] Nov 01 16:39:23.916447 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:39:23.916421 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e6beffd_779a_4c53_a78d_a5beda2ba056.slice/crio-808101e4b1ba3ba1e2f9bd4df43eb3e2f4948efaeed87b85a8b1b8f72467cb3c WatchSource:0}: Error finding container 808101e4b1ba3ba1e2f9bd4df43eb3e2f4948efaeed87b85a8b1b8f72467cb3c: Status 404 returned error can't find the container with id 808101e4b1ba3ba1e2f9bd4df43eb3e2f4948efaeed87b85a8b1b8f72467cb3c Nov 01 16:39:24.370094 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:39:24.370049 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:39:24.552972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:24.552933 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" event={"ID":"6e6beffd-779a-4c53-a78d-a5beda2ba056","Type":"ContainerStarted","Data":"54b7f7c8fde255e8fe9b14025a8611c0f10b8076cd344ed6cb496fb5851bc62a"} Nov 01 16:39:24.553173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:24.553000 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" event={"ID":"6e6beffd-779a-4c53-a78d-a5beda2ba056","Type":"ContainerStarted","Data":"808101e4b1ba3ba1e2f9bd4df43eb3e2f4948efaeed87b85a8b1b8f72467cb3c"} Nov 01 16:39:24.569274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:24.569218 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" podStartSLOduration=1.56919971 podStartE2EDuration="1.56919971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:39:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:39:24.56895322 +0000 UTC m=+3102.272119157" watchObservedRunningTime="2026-11-01 16:39:24.56919971 +0000 UTC m=+3102.272365647" Nov 01 16:39:25.560168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:25.560133 4573 generic.go:356] "Generic (PLEG): container finished" podID="6e6beffd-779a-4c53-a78d-a5beda2ba056" containerID="54b7f7c8fde255e8fe9b14025a8611c0f10b8076cd344ed6cb496fb5851bc62a" exitCode=0 Nov 01 16:39:25.560572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:25.560214 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" event={"ID":"6e6beffd-779a-4c53-a78d-a5beda2ba056","Type":"ContainerDied","Data":"54b7f7c8fde255e8fe9b14025a8611c0f10b8076cd344ed6cb496fb5851bc62a"} Nov 01 16:39:26.840438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:26.840355 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:26.918191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:26.918156 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-xks-ca\") pod \"6e6beffd-779a-4c53-a78d-a5beda2ba056\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " Nov 01 16:39:26.918379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:26.918248 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e6beffd-779a-4c53-a78d-a5beda2ba056-kube-api-access-fmcrb\" (UniqueName: \"kubernetes.io/projected/6e6beffd-779a-4c53-a78d-a5beda2ba056-kube-api-access-fmcrb\") pod \"6e6beffd-779a-4c53-a78d-a5beda2ba056\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " Nov 01 16:39:26.918379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:26.918330 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-openshift-ca\") pod \"6e6beffd-779a-4c53-a78d-a5beda2ba056\" (UID: \"6e6beffd-779a-4c53-a78d-a5beda2ba056\") " Nov 01 16:39:26.918685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:26.918662 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-openshift-ca" pod "6e6beffd-779a-4c53-a78d-a5beda2ba056" (UID: "6e6beffd-779a-4c53-a78d-a5beda2ba056"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:39:26.920271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:26.920238 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e6beffd-779a-4c53-a78d-a5beda2ba056-kube-api-access-fmcrb" pod "6e6beffd-779a-4c53-a78d-a5beda2ba056" (UID: "6e6beffd-779a-4c53-a78d-a5beda2ba056"). InnerVolumeSpecName "kube-api-access-fmcrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:39:26.920271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:26.920262 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-xks-ca" pod "6e6beffd-779a-4c53-a78d-a5beda2ba056" (UID: "6e6beffd-779a-4c53-a78d-a5beda2ba056"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:39:27.019661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:27.019638 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:27.019661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:27.019660 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmcrb\" (UniqueName: \"kubernetes.io/projected/6e6beffd-779a-4c53-a78d-a5beda2ba056-kube-api-access-fmcrb\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:27.019852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:27.019672 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e6beffd-779a-4c53-a78d-a5beda2ba056-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:27.581632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:27.581597 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" Nov 01 16:39:27.581845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:27.581602 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm" event={"ID":"6e6beffd-779a-4c53-a78d-a5beda2ba056","Type":"ContainerDied","Data":"808101e4b1ba3ba1e2f9bd4df43eb3e2f4948efaeed87b85a8b1b8f72467cb3c"} Nov 01 16:39:27.581845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:27.581728 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="808101e4b1ba3ba1e2f9bd4df43eb3e2f4948efaeed87b85a8b1b8f72467cb3c" Nov 01 16:39:32.727286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:32.727245 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:39:34.153066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:34.153028 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:39:39.166121 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:39.166078 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:39:44.092512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.092477 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t"] Nov 01 16:39:44.092925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.092911 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6e6beffd-779a-4c53-a78d-a5beda2ba056" containerName="revoke-keys" Nov 01 16:39:44.092983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.092926 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e6beffd-779a-4c53-a78d-a5beda2ba056" containerName="revoke-keys" Nov 01 16:39:44.093043 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.093033 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e6beffd-779a-4c53-a78d-a5beda2ba056" containerName="revoke-keys" Nov 01 16:39:44.096026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.096008 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.102387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.102340 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:39:44.102707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.102690 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:39:44.105368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.105339 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t"] Nov 01 16:39:44.157665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.157634 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:39:44.205898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.205859 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwlrx\" (UniqueName: \"kubernetes.io/projected/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-kube-api-access-nwlrx\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.206125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.205930 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.206125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.205959 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.306847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.306808 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwlrx\" (UniqueName: \"kubernetes.io/projected/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-kube-api-access-nwlrx\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.307053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.306892 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.307053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.306925 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.307599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.307578 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.309256 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.309241 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.314955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.314932 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwlrx\" (UniqueName: \"kubernetes.io/projected/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-kube-api-access-nwlrx\") pod \"maas-api-revoke-sub-ae594979-2pl6t\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.411289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.411259 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:44.691984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:44.691951 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t"] Nov 01 16:39:44.693140 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:39:44.693114 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffde061e_8126_4ba0_b8fb_0f2dc51b7411.slice/crio-dc890f2c8c1198f9b22918496e9c3e347ec27b6357faa168aaf8555878cbd349 WatchSource:0}: Error finding container dc890f2c8c1198f9b22918496e9c3e347ec27b6357faa168aaf8555878cbd349: Status 404 returned error can't find the container with id dc890f2c8c1198f9b22918496e9c3e347ec27b6357faa168aaf8555878cbd349 Nov 01 16:39:45.689366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:45.689332 4573 generic.go:356] "Generic (PLEG): container finished" podID="ffde061e-8126-4ba0-b8fb-0f2dc51b7411" containerID="267fe5f87888238f906169e551b2dd67eb201a61c79310b3caddaf2de3d7e9bb" exitCode=0 Nov 01 16:39:45.689366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:45.689370 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" event={"ID":"ffde061e-8126-4ba0-b8fb-0f2dc51b7411","Type":"ContainerDied","Data":"267fe5f87888238f906169e551b2dd67eb201a61c79310b3caddaf2de3d7e9bb"} Nov 01 16:39:45.689826 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:45.689417 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" event={"ID":"ffde061e-8126-4ba0-b8fb-0f2dc51b7411","Type":"ContainerStarted","Data":"dc890f2c8c1198f9b22918496e9c3e347ec27b6357faa168aaf8555878cbd349"} Nov 01 16:39:46.829092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:46.829068 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:46.931183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:46.931152 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-xks-ca\") pod \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " Nov 01 16:39:46.931322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:46.931208 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-kube-api-access-nwlrx\" (UniqueName: \"kubernetes.io/projected/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-kube-api-access-nwlrx\") pod \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " Nov 01 16:39:46.931384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:46.931319 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-openshift-ca\") pod \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\" (UID: \"ffde061e-8126-4ba0-b8fb-0f2dc51b7411\") " Nov 01 16:39:46.931711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:46.931687 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-openshift-ca" pod "ffde061e-8126-4ba0-b8fb-0f2dc51b7411" (UID: "ffde061e-8126-4ba0-b8fb-0f2dc51b7411"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:39:46.933240 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:46.933218 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-xks-ca" pod "ffde061e-8126-4ba0-b8fb-0f2dc51b7411" (UID: "ffde061e-8126-4ba0-b8fb-0f2dc51b7411"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:39:46.933338 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:46.933300 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-kube-api-access-nwlrx" pod "ffde061e-8126-4ba0-b8fb-0f2dc51b7411" (UID: "ffde061e-8126-4ba0-b8fb-0f2dc51b7411"). InnerVolumeSpecName "kube-api-access-nwlrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:39:47.032221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:47.032157 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:47.032221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:47.032184 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwlrx\" (UniqueName: \"kubernetes.io/projected/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-kube-api-access-nwlrx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:47.032221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:47.032195 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffde061e-8126-4ba0-b8fb-0f2dc51b7411-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:39:47.702428 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:47.702400 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" Nov 01 16:39:47.702638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:47.702441 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t" event={"ID":"ffde061e-8126-4ba0-b8fb-0f2dc51b7411","Type":"ContainerDied","Data":"dc890f2c8c1198f9b22918496e9c3e347ec27b6357faa168aaf8555878cbd349"} Nov 01 16:39:47.702638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:47.702488 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc890f2c8c1198f9b22918496e9c3e347ec27b6357faa168aaf8555878cbd349" Nov 01 16:39:48.243742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:48.243709 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:39:50.064284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.064253 4573 scope.go:122] "RemoveContainer" containerID="64e6beffd4ade3b0aa5b952878a63fb4ca70dd09f231e8e424652755b59ea125" Nov 01 16:39:50.074027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.073998 4573 scope.go:122] "RemoveContainer" containerID="675b1e1f0a5d52b8586284be4a32bbb6a250460ed4301e91bc1ab34058d1e7aa" Nov 01 16:39:50.083027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.083004 4573 scope.go:122] "RemoveContainer" containerID="7596b45836069d3c939ddd0e819db86ae7c75f65c111194cc646ecdabac5eeee" Nov 01 16:39:50.091565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.091544 4573 scope.go:122] "RemoveContainer" containerID="1df6001a76d3e3aa6bce9cda586df0f9ef9f54044d6fb6f388df8109f032b580" Nov 01 16:39:50.099397 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.099373 4573 scope.go:122] "RemoveContainer" containerID="1fdb2220044375484ae10740e434873c983c3e98a28ae82bc4e288f4fcc74533" Nov 01 16:39:50.107049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.107027 4573 scope.go:122] "RemoveContainer" containerID="f9286f5431abdabd1625990a3139f8a096ad2252862924c1fcb1e04731356d05" Nov 01 16:39:50.114551 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.114533 4573 scope.go:122] "RemoveContainer" containerID="ebe50d6c5cc80b64b6018a2a623a3cf2ae5b6d942f12dd095d40957fefcc61aa" Nov 01 16:39:50.122094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.122079 4573 scope.go:122] "RemoveContainer" containerID="2eaf3ad402d96ebb539bba781306a154c5b31a0168b251d88a8fdc032e0742cd" Nov 01 16:39:50.129446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:50.129430 4573 scope.go:122] "RemoveContainer" containerID="197a004a771ee8894cea1a15bf22f3c6514d8d1739cf98179966e977d69dd805" Nov 01 16:39:54.363244 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:39:54.363208 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:39:54.877438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:54.877403 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:39:58.937494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:39:58.937450 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:40:02.393547 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.393512 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s"] Nov 01 16:40:02.393994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.393969 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ffde061e-8126-4ba0-b8fb-0f2dc51b7411" containerName="revoke-keys" Nov 01 16:40:02.393994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.393991 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffde061e-8126-4ba0-b8fb-0f2dc51b7411" containerName="revoke-keys" Nov 01 16:40:02.394102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.394091 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffde061e-8126-4ba0-b8fb-0f2dc51b7411" containerName="revoke-keys" Nov 01 16:40:02.399824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.399800 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.416676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.416638 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:40:02.416901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.416704 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:40:02.420441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.420405 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s"] Nov 01 16:40:02.482925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.482885 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdf4\" (UniqueName: \"kubernetes.io/projected/eb715982-66ce-4ea8-8d53-dcfe3bf487db-kube-api-access-frdf4\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.482925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.482927 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.483211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.482948 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.584082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.584034 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdf4\" (UniqueName: \"kubernetes.io/projected/eb715982-66ce-4ea8-8d53-dcfe3bf487db-kube-api-access-frdf4\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.584082 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.584088 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.584393 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.584118 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.584925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.584899 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.586638 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.586615 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.591735 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.591706 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdf4\" (UniqueName: \"kubernetes.io/projected/eb715982-66ce-4ea8-8d53-dcfe3bf487db-kube-api-access-frdf4\") pod \"maas-api-revoke-sub-a4aef7d7-tfz4s\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.714804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.714692 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:02.991667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:02.991641 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s"] Nov 01 16:40:02.993448 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:40:02.993415 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb715982_66ce_4ea8_8d53_dcfe3bf487db.slice/crio-24e6a4f76bf0424d1e3f3343f122e8ef4092763f77678905a7a4d5617801d17a WatchSource:0}: Error finding container 24e6a4f76bf0424d1e3f3343f122e8ef4092763f77678905a7a4d5617801d17a: Status 404 returned error can't find the container with id 24e6a4f76bf0424d1e3f3343f122e8ef4092763f77678905a7a4d5617801d17a Nov 01 16:40:03.806440 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:03.806412 4573 generic.go:356] "Generic (PLEG): container finished" podID="eb715982-66ce-4ea8-8d53-dcfe3bf487db" containerID="f4361abd14e230124a8ee330938ddd6b94bb4cd102d3290d5fb0046fcd8641b4" exitCode=0 Nov 01 16:40:03.806790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:03.806490 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" event={"ID":"eb715982-66ce-4ea8-8d53-dcfe3bf487db","Type":"ContainerDied","Data":"f4361abd14e230124a8ee330938ddd6b94bb4cd102d3290d5fb0046fcd8641b4"} Nov 01 16:40:03.806790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:03.806568 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" event={"ID":"eb715982-66ce-4ea8-8d53-dcfe3bf487db","Type":"ContainerStarted","Data":"24e6a4f76bf0424d1e3f3343f122e8ef4092763f77678905a7a4d5617801d17a"} Nov 01 16:40:04.946568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:04.946543 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:05.007467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.007434 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb715982-66ce-4ea8-8d53-dcfe3bf487db-kube-api-access-frdf4\" (UniqueName: \"kubernetes.io/projected/eb715982-66ce-4ea8-8d53-dcfe3bf487db-kube-api-access-frdf4\") pod \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " Nov 01 16:40:05.007655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.007474 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-openshift-ca\") pod \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " Nov 01 16:40:05.007655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.007510 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-xks-ca\") pod \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\" (UID: \"eb715982-66ce-4ea8-8d53-dcfe3bf487db\") " Nov 01 16:40:05.007994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.007972 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-openshift-ca" pod "eb715982-66ce-4ea8-8d53-dcfe3bf487db" (UID: "eb715982-66ce-4ea8-8d53-dcfe3bf487db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:40:05.009602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.009581 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-xks-ca" pod "eb715982-66ce-4ea8-8d53-dcfe3bf487db" (UID: "eb715982-66ce-4ea8-8d53-dcfe3bf487db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:40:05.009699 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.009677 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb715982-66ce-4ea8-8d53-dcfe3bf487db-kube-api-access-frdf4" pod "eb715982-66ce-4ea8-8d53-dcfe3bf487db" (UID: "eb715982-66ce-4ea8-8d53-dcfe3bf487db"). InnerVolumeSpecName "kube-api-access-frdf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:40:05.108870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.108843 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:05.108870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.108869 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdf4\" (UniqueName: \"kubernetes.io/projected/eb715982-66ce-4ea8-8d53-dcfe3bf487db-kube-api-access-frdf4\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:05.109048 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.108880 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb715982-66ce-4ea8-8d53-dcfe3bf487db-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:05.819556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.819521 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" event={"ID":"eb715982-66ce-4ea8-8d53-dcfe3bf487db","Type":"ContainerDied","Data":"24e6a4f76bf0424d1e3f3343f122e8ef4092763f77678905a7a4d5617801d17a"} Nov 01 16:40:05.819556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.819547 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s" Nov 01 16:40:05.819879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:05.819573 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24e6a4f76bf0424d1e3f3343f122e8ef4092763f77678905a7a4d5617801d17a" Nov 01 16:40:13.164424 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:13.164380 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:40:18.618536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:18.618488 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:40:21.152770 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.152726 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7"] Nov 01 16:40:21.153237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.153218 4573 cpu_manager.go:414] "containerMap: removing container" podUID="eb715982-66ce-4ea8-8d53-dcfe3bf487db" containerName="revoke-keys" Nov 01 16:40:21.153237 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.153232 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb715982-66ce-4ea8-8d53-dcfe3bf487db" containerName="revoke-keys" Nov 01 16:40:21.153363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.153346 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb715982-66ce-4ea8-8d53-dcfe3bf487db" containerName="revoke-keys" Nov 01 16:40:21.156856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.156832 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.160933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.160897 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:40:21.161063 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.161010 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:40:21.163379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.163354 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7"] Nov 01 16:40:21.305815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.305770 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2llf8\" (UniqueName: \"kubernetes.io/projected/2b617307-2229-4e9d-a9d5-72140bc298df-kube-api-access-2llf8\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.305815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.305814 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.306055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.305950 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.407113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.407033 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2llf8\" (UniqueName: \"kubernetes.io/projected/2b617307-2229-4e9d-a9d5-72140bc298df-kube-api-access-2llf8\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.407113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.407070 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.407301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.407123 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.407898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.407879 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.409456 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.409433 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.415548 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.415528 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2llf8\" (UniqueName: \"kubernetes.io/projected/2b617307-2229-4e9d-a9d5-72140bc298df-kube-api-access-2llf8\") pod \"maas-api-revoke-sub-79805ca9-ln7g7\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.473076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.473051 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:21.751341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.751314 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7"] Nov 01 16:40:21.752882 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:40:21.752853 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b617307_2229_4e9d_a9d5_72140bc298df.slice/crio-69ec850fd419d6408a85ce03ea8c002a657e0a480523e1f4f7c38ac2d16bfc4c WatchSource:0}: Error finding container 69ec850fd419d6408a85ce03ea8c002a657e0a480523e1f4f7c38ac2d16bfc4c: Status 404 returned error can't find the container with id 69ec850fd419d6408a85ce03ea8c002a657e0a480523e1f4f7c38ac2d16bfc4c Nov 01 16:40:21.948116 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:21.948083 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" event={"ID":"2b617307-2229-4e9d-a9d5-72140bc298df","Type":"ContainerStarted","Data":"69ec850fd419d6408a85ce03ea8c002a657e0a480523e1f4f7c38ac2d16bfc4c"} Nov 01 16:40:22.955345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:22.955311 4573 generic.go:356] "Generic (PLEG): container finished" podID="2b617307-2229-4e9d-a9d5-72140bc298df" containerID="9868e7756b9e3ba66a046158d5c506a6f5c4ba3a13defc1ed11fa939480991e9" exitCode=0 Nov 01 16:40:22.955767 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:22.955352 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" event={"ID":"2b617307-2229-4e9d-a9d5-72140bc298df","Type":"ContainerDied","Data":"9868e7756b9e3ba66a046158d5c506a6f5c4ba3a13defc1ed11fa939480991e9"} Nov 01 16:40:24.090784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.090740 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:24.235211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.235130 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-openshift-ca\") pod \"2b617307-2229-4e9d-a9d5-72140bc298df\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " Nov 01 16:40:24.235379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.235213 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b617307-2229-4e9d-a9d5-72140bc298df-kube-api-access-2llf8\" (UniqueName: \"kubernetes.io/projected/2b617307-2229-4e9d-a9d5-72140bc298df-kube-api-access-2llf8\") pod \"2b617307-2229-4e9d-a9d5-72140bc298df\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " Nov 01 16:40:24.235379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.235315 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-xks-ca\") pod \"2b617307-2229-4e9d-a9d5-72140bc298df\" (UID: \"2b617307-2229-4e9d-a9d5-72140bc298df\") " Nov 01 16:40:24.235574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.235549 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-openshift-ca" pod "2b617307-2229-4e9d-a9d5-72140bc298df" (UID: "2b617307-2229-4e9d-a9d5-72140bc298df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:40:24.235664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.235650 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:24.237172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.237147 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b617307-2229-4e9d-a9d5-72140bc298df-kube-api-access-2llf8" pod "2b617307-2229-4e9d-a9d5-72140bc298df" (UID: "2b617307-2229-4e9d-a9d5-72140bc298df"). InnerVolumeSpecName "kube-api-access-2llf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:40:24.237266 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.237197 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-xks-ca" pod "2b617307-2229-4e9d-a9d5-72140bc298df" (UID: "2b617307-2229-4e9d-a9d5-72140bc298df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:40:24.336925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.336891 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2llf8\" (UniqueName: \"kubernetes.io/projected/2b617307-2229-4e9d-a9d5-72140bc298df-kube-api-access-2llf8\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:24.336925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.336923 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b617307-2229-4e9d-a9d5-72140bc298df-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:24.363065 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:40:24.363024 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:40:24.968493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.968465 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" Nov 01 16:40:24.968493 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.968480 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7" event={"ID":"2b617307-2229-4e9d-a9d5-72140bc298df","Type":"ContainerDied","Data":"69ec850fd419d6408a85ce03ea8c002a657e0a480523e1f4f7c38ac2d16bfc4c"} Nov 01 16:40:24.968713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:24.968522 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ec850fd419d6408a85ce03ea8c002a657e0a480523e1f4f7c38ac2d16bfc4c" Nov 01 16:40:32.040191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:32.040160 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:40:32.777667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:32.777631 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:40:36.029775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:36.029463 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:40:38.843295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.843257 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs"] Nov 01 16:40:38.843691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.843679 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2b617307-2229-4e9d-a9d5-72140bc298df" containerName="revoke-keys" Nov 01 16:40:38.843691 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.843690 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b617307-2229-4e9d-a9d5-72140bc298df" containerName="revoke-keys" Nov 01 16:40:38.843811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.843801 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b617307-2229-4e9d-a9d5-72140bc298df" containerName="revoke-keys" Nov 01 16:40:38.847287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.847272 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:38.853269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.853173 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:40:38.853269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.853241 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:40:38.855930 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.855438 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs"] Nov 01 16:40:38.980988 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.980943 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c594\" (UniqueName: \"kubernetes.io/projected/b2b7f712-7b38-480d-8bbb-5de26ace1266-kube-api-access-9c594\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:38.981170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.981005 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:38.981170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:38.981124 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:39.082578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:39.082543 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:39.082768 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:39.082602 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:39.082768 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:39.082668 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c594\" (UniqueName: \"kubernetes.io/projected/b2b7f712-7b38-480d-8bbb-5de26ace1266-kube-api-access-9c594\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:39.083313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:39.083287 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:39.085023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:39.085001 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:39.089702 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:39.089680 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c594\" (UniqueName: \"kubernetes.io/projected/b2b7f712-7b38-480d-8bbb-5de26ace1266-kube-api-access-9c594\") pod \"maas-api-revoke-sub-0b7e8edb-7dkqs\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:39.161915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:39.161893 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:39.443091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:39.443058 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs"] Nov 01 16:40:39.444810 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:40:39.444737 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2b7f712_7b38_480d_8bbb_5de26ace1266.slice/crio-18ea1c8ed88f23f98b987be2ddc5abb6b14366e949692fafbff4242dfffa59c3 WatchSource:0}: Error finding container 18ea1c8ed88f23f98b987be2ddc5abb6b14366e949692fafbff4242dfffa59c3: Status 404 returned error can't find the container with id 18ea1c8ed88f23f98b987be2ddc5abb6b14366e949692fafbff4242dfffa59c3 Nov 01 16:40:40.065134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:40.065034 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" event={"ID":"b2b7f712-7b38-480d-8bbb-5de26ace1266","Type":"ContainerStarted","Data":"87e6ddfb582e9698d215659a0fbfa2a12cedcb9eaed84bb9ce7d066629900c2b"} Nov 01 16:40:40.065134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:40.065111 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" event={"ID":"b2b7f712-7b38-480d-8bbb-5de26ace1266","Type":"ContainerStarted","Data":"18ea1c8ed88f23f98b987be2ddc5abb6b14366e949692fafbff4242dfffa59c3"} Nov 01 16:40:40.080908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:40.080854 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" podStartSLOduration=2.0808374020000002 podStartE2EDuration="2.080837402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:40:40.080152345 +0000 UTC m=+3177.783318284" watchObservedRunningTime="2026-11-01 16:40:40.080837402 +0000 UTC m=+3177.784003362" Nov 01 16:40:41.073056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:41.073022 4573 generic.go:356] "Generic (PLEG): container finished" podID="b2b7f712-7b38-480d-8bbb-5de26ace1266" containerID="87e6ddfb582e9698d215659a0fbfa2a12cedcb9eaed84bb9ce7d066629900c2b" exitCode=0 Nov 01 16:40:41.073427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:41.073103 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" event={"ID":"b2b7f712-7b38-480d-8bbb-5de26ace1266","Type":"ContainerDied","Data":"87e6ddfb582e9698d215659a0fbfa2a12cedcb9eaed84bb9ce7d066629900c2b"} Nov 01 16:40:42.223076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.223044 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:42.321345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.321314 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-openshift-ca\") pod \"b2b7f712-7b38-480d-8bbb-5de26ace1266\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " Nov 01 16:40:42.321518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.321386 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-xks-ca\") pod \"b2b7f712-7b38-480d-8bbb-5de26ace1266\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " Nov 01 16:40:42.321518 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.321415 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2b7f712-7b38-480d-8bbb-5de26ace1266-kube-api-access-9c594\" (UniqueName: \"kubernetes.io/projected/b2b7f712-7b38-480d-8bbb-5de26ace1266-kube-api-access-9c594\") pod \"b2b7f712-7b38-480d-8bbb-5de26ace1266\" (UID: \"b2b7f712-7b38-480d-8bbb-5de26ace1266\") " Nov 01 16:40:42.321737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.321714 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-openshift-ca" pod "b2b7f712-7b38-480d-8bbb-5de26ace1266" (UID: "b2b7f712-7b38-480d-8bbb-5de26ace1266"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:40:42.323534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.323507 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-xks-ca" pod "b2b7f712-7b38-480d-8bbb-5de26ace1266" (UID: "b2b7f712-7b38-480d-8bbb-5de26ace1266"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:40:42.323668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.323588 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2b7f712-7b38-480d-8bbb-5de26ace1266-kube-api-access-9c594" pod "b2b7f712-7b38-480d-8bbb-5de26ace1266" (UID: "b2b7f712-7b38-480d-8bbb-5de26ace1266"). InnerVolumeSpecName "kube-api-access-9c594". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:40:42.422344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.422316 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:42.422344 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.422343 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2b7f712-7b38-480d-8bbb-5de26ace1266-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:42.422506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:42.422353 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c594\" (UniqueName: \"kubernetes.io/projected/b2b7f712-7b38-480d-8bbb-5de26ace1266-kube-api-access-9c594\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:43.086026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:43.085998 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" Nov 01 16:40:43.086201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:43.086021 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs" event={"ID":"b2b7f712-7b38-480d-8bbb-5de26ace1266","Type":"ContainerDied","Data":"18ea1c8ed88f23f98b987be2ddc5abb6b14366e949692fafbff4242dfffa59c3"} Nov 01 16:40:43.086201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:43.086076 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18ea1c8ed88f23f98b987be2ddc5abb6b14366e949692fafbff4242dfffa59c3" Nov 01 16:40:45.021491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:45.021453 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck"] Nov 01 16:40:45.023704 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:45.023677 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-472dee5d-gzbck"] Nov 01 16:40:46.456880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:46.456846 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d04b34e-2bfd-4588-b268-6c98be4c260b" path="/var/lib/kubelet/pods/2d04b34e-2bfd-4588-b268-6c98be4c260b/volumes" Nov 01 16:40:49.499158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.499118 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4"] Nov 01 16:40:49.499795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.499778 4573 cpu_manager.go:414] "containerMap: removing container" podUID="b2b7f712-7b38-480d-8bbb-5de26ace1266" containerName="revoke-keys" Nov 01 16:40:49.499849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.499797 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2b7f712-7b38-480d-8bbb-5de26ace1266" containerName="revoke-keys" Nov 01 16:40:49.499918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.499907 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2b7f712-7b38-480d-8bbb-5de26ace1266" containerName="revoke-keys" Nov 01 16:40:49.503451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.503434 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.507435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.507404 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:40:49.507606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.507572 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:40:49.509029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.509004 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4"] Nov 01 16:40:49.592224 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.592188 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.592395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.592235 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.592486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.592394 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtq2h\" (UniqueName: \"kubernetes.io/projected/617cad46-d825-45d0-981d-b6b1bf8e9f7d-kube-api-access-qtq2h\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.654276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.654239 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:40:49.693579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.693549 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.693790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.693595 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.693790 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.693672 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtq2h\" (UniqueName: \"kubernetes.io/projected/617cad46-d825-45d0-981d-b6b1bf8e9f7d-kube-api-access-qtq2h\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.694417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.694394 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.696012 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.695988 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.701096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.701076 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtq2h\" (UniqueName: \"kubernetes.io/projected/617cad46-d825-45d0-981d-b6b1bf8e9f7d-kube-api-access-qtq2h\") pod \"maas-api-revoke-sub-3d4156f1-p4vd4\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:49.821739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:49.821658 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:50.111491 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:50.111448 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4"] Nov 01 16:40:50.115346 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:40:50.115317 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617cad46_d825_45d0_981d_b6b1bf8e9f7d.slice/crio-b738410591e1d46093f05fa4340605d2d1b8467b32532bfff37bcdf1ef6e1823 WatchSource:0}: Error finding container b738410591e1d46093f05fa4340605d2d1b8467b32532bfff37bcdf1ef6e1823: Status 404 returned error can't find the container with id b738410591e1d46093f05fa4340605d2d1b8467b32532bfff37bcdf1ef6e1823 Nov 01 16:40:50.126575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:50.126544 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" event={"ID":"617cad46-d825-45d0-981d-b6b1bf8e9f7d","Type":"ContainerStarted","Data":"b738410591e1d46093f05fa4340605d2d1b8467b32532bfff37bcdf1ef6e1823"} Nov 01 16:40:50.182985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:50.182963 4573 scope.go:122] "RemoveContainer" containerID="db195488edac1931324e5f21e5778c8b8a32bef95eadc8fd2779f0de7de817bd" Nov 01 16:40:50.191144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:50.191127 4573 scope.go:122] "RemoveContainer" containerID="d9c8d44656ab9942b560d664646beb6e085ddcf037dce5f97dc1d09ec3225b84" Nov 01 16:40:50.201442 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:50.201363 4573 scope.go:122] "RemoveContainer" containerID="4041c25e4d7cf5119c6771f1f38c97519289b1b98e2fda78dcc7dd27b975828d" Nov 01 16:40:50.209109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:50.209091 4573 scope.go:122] "RemoveContainer" containerID="188ab1fa1f7cfb6e5a632b3265a4bcd5cdb866f1bb78643ccddb776605c4b9bb" Nov 01 16:40:51.133985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:51.133945 4573 generic.go:356] "Generic (PLEG): container finished" podID="617cad46-d825-45d0-981d-b6b1bf8e9f7d" containerID="4370146b6f4a38965c2e365d69193b167ed055ea5d97061d495534711eea570c" exitCode=0 Nov 01 16:40:51.134406 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:51.133989 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" event={"ID":"617cad46-d825-45d0-981d-b6b1bf8e9f7d","Type":"ContainerDied","Data":"4370146b6f4a38965c2e365d69193b167ed055ea5d97061d495534711eea570c"} Nov 01 16:40:52.271042 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.271017 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:52.319550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.319513 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-openshift-ca\") pod \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " Nov 01 16:40:52.319721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.319618 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-xks-ca\") pod \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " Nov 01 16:40:52.319797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.319717 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617cad46-d825-45d0-981d-b6b1bf8e9f7d-kube-api-access-qtq2h\" (UniqueName: \"kubernetes.io/projected/617cad46-d825-45d0-981d-b6b1bf8e9f7d-kube-api-access-qtq2h\") pod \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\" (UID: \"617cad46-d825-45d0-981d-b6b1bf8e9f7d\") " Nov 01 16:40:52.320035 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.320010 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-openshift-ca" pod "617cad46-d825-45d0-981d-b6b1bf8e9f7d" (UID: "617cad46-d825-45d0-981d-b6b1bf8e9f7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:40:52.321853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.321829 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617cad46-d825-45d0-981d-b6b1bf8e9f7d-kube-api-access-qtq2h" pod "617cad46-d825-45d0-981d-b6b1bf8e9f7d" (UID: "617cad46-d825-45d0-981d-b6b1bf8e9f7d"). InnerVolumeSpecName "kube-api-access-qtq2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:40:52.321945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.321888 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-xks-ca" pod "617cad46-d825-45d0-981d-b6b1bf8e9f7d" (UID: "617cad46-d825-45d0-981d-b6b1bf8e9f7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:40:52.420786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.420683 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:52.420786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.420711 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtq2h\" (UniqueName: \"kubernetes.io/projected/617cad46-d825-45d0-981d-b6b1bf8e9f7d-kube-api-access-qtq2h\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:52.420786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:52.420726 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/617cad46-d825-45d0-981d-b6b1bf8e9f7d-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:40:53.146218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:53.146189 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" Nov 01 16:40:53.146395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:53.146223 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4" event={"ID":"617cad46-d825-45d0-981d-b6b1bf8e9f7d","Type":"ContainerDied","Data":"b738410591e1d46093f05fa4340605d2d1b8467b32532bfff37bcdf1ef6e1823"} Nov 01 16:40:53.146395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:40:53.146267 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b738410591e1d46093f05fa4340605d2d1b8467b32532bfff37bcdf1ef6e1823" Nov 01 16:40:54.363951 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:40:54.363876 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:41:00.278355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:00.278318 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:41:04.582159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:04.582122 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:41:07.519077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.519043 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4"] Nov 01 16:41:07.523800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.523769 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p4vd4"] Nov 01 16:41:07.540962 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.540929 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn"] Nov 01 16:41:07.541355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.541341 4573 cpu_manager.go:414] "containerMap: removing container" podUID="617cad46-d825-45d0-981d-b6b1bf8e9f7d" containerName="revoke-keys" Nov 01 16:41:07.541355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.541355 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617cad46-d825-45d0-981d-b6b1bf8e9f7d" containerName="revoke-keys" Nov 01 16:41:07.541451 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.541415 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="617cad46-d825-45d0-981d-b6b1bf8e9f7d" containerName="revoke-keys" Nov 01 16:41:07.547588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.547560 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.552445 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.552422 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:41:07.553927 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.553902 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:41:07.554696 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.554596 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn"] Nov 01 16:41:07.591598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.591566 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.591784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.591604 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb76q\" (UniqueName: \"kubernetes.io/projected/d590df74-4e77-4a8c-a863-e97c2a26fee3-kube-api-access-kb76q\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.591784 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.591740 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.692835 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.692798 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.693024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.692874 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.693024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.692903 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb76q\" (UniqueName: \"kubernetes.io/projected/d590df74-4e77-4a8c-a863-e97c2a26fee3-kube-api-access-kb76q\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.693545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.693522 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.695204 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.695180 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.702092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.702070 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb76q\" (UniqueName: \"kubernetes.io/projected/d590df74-4e77-4a8c-a863-e97c2a26fee3-kube-api-access-kb76q\") pod \"maas-api-revoke-sub-3d4156f1-2h2vn\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:07.868855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:07.868822 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:08.143300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:08.143270 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn"] Nov 01 16:41:08.145900 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:41:08.145867 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd590df74_4e77_4a8c_a863_e97c2a26fee3.slice/crio-ac103b58560684cd6000b0975f0259bdf26c358c29c87b72cee4520668b90821 WatchSource:0}: Error finding container ac103b58560684cd6000b0975f0259bdf26c358c29c87b72cee4520668b90821: Status 404 returned error can't find the container with id ac103b58560684cd6000b0975f0259bdf26c358c29c87b72cee4520668b90821 Nov 01 16:41:08.239444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:08.239414 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" event={"ID":"d590df74-4e77-4a8c-a863-e97c2a26fee3","Type":"ContainerStarted","Data":"ac103b58560684cd6000b0975f0259bdf26c358c29c87b72cee4520668b90821"} Nov 01 16:41:08.457411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:08.457381 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="617cad46-d825-45d0-981d-b6b1bf8e9f7d" path="/var/lib/kubelet/pods/617cad46-d825-45d0-981d-b6b1bf8e9f7d/volumes" Nov 01 16:41:09.247373 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:09.247285 4573 generic.go:356] "Generic (PLEG): container finished" podID="d590df74-4e77-4a8c-a863-e97c2a26fee3" containerID="d3e72cab8462cf7d2a51d89b3cdc6bc2c321b54214be3d69c3f0aef3d4298816" exitCode=0 Nov 01 16:41:09.247809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:09.247362 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" event={"ID":"d590df74-4e77-4a8c-a863-e97c2a26fee3","Type":"ContainerDied","Data":"d3e72cab8462cf7d2a51d89b3cdc6bc2c321b54214be3d69c3f0aef3d4298816"} Nov 01 16:41:10.393306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.393279 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:10.519396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.519309 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-openshift-ca\") pod \"d590df74-4e77-4a8c-a863-e97c2a26fee3\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " Nov 01 16:41:10.519396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.519375 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-xks-ca\") pod \"d590df74-4e77-4a8c-a863-e97c2a26fee3\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " Nov 01 16:41:10.519692 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.519491 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d590df74-4e77-4a8c-a863-e97c2a26fee3-kube-api-access-kb76q\" (UniqueName: \"kubernetes.io/projected/d590df74-4e77-4a8c-a863-e97c2a26fee3-kube-api-access-kb76q\") pod \"d590df74-4e77-4a8c-a863-e97c2a26fee3\" (UID: \"d590df74-4e77-4a8c-a863-e97c2a26fee3\") " Nov 01 16:41:10.519796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.519745 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-openshift-ca" pod "d590df74-4e77-4a8c-a863-e97c2a26fee3" (UID: "d590df74-4e77-4a8c-a863-e97c2a26fee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:41:10.519935 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.519911 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:10.521508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.521477 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-xks-ca" pod "d590df74-4e77-4a8c-a863-e97c2a26fee3" (UID: "d590df74-4e77-4a8c-a863-e97c2a26fee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:41:10.521615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.521545 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d590df74-4e77-4a8c-a863-e97c2a26fee3-kube-api-access-kb76q" pod "d590df74-4e77-4a8c-a863-e97c2a26fee3" (UID: "d590df74-4e77-4a8c-a863-e97c2a26fee3"). InnerVolumeSpecName "kube-api-access-kb76q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:41:10.621306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.621272 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d590df74-4e77-4a8c-a863-e97c2a26fee3-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:10.621306 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:10.621300 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb76q\" (UniqueName: \"kubernetes.io/projected/d590df74-4e77-4a8c-a863-e97c2a26fee3-kube-api-access-kb76q\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:11.260115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:11.260077 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" event={"ID":"d590df74-4e77-4a8c-a863-e97c2a26fee3","Type":"ContainerDied","Data":"ac103b58560684cd6000b0975f0259bdf26c358c29c87b72cee4520668b90821"} Nov 01 16:41:11.260309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:11.260122 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn" Nov 01 16:41:11.260309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:11.260133 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac103b58560684cd6000b0975f0259bdf26c358c29c87b72cee4520668b90821" Nov 01 16:41:18.344280 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:18.344241 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:41:21.468740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:21.468706 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:41:24.363046 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:41:24.363006 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:41:28.573745 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:28.573705 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:41:31.032905 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:31.032857 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4"] Nov 01 16:41:31.036151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:31.036114 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-mhdf4"] Nov 01 16:41:32.456260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:32.456230 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa96599-7fce-4ee1-849a-0a212fdccc0d" path="/var/lib/kubelet/pods/6aa96599-7fce-4ee1-849a-0a212fdccc0d/volumes" Nov 01 16:41:34.093898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:34.093837 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:41:36.124715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.124674 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb"] Nov 01 16:41:36.125278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.125188 4573 cpu_manager.go:414] "containerMap: removing container" podUID="d590df74-4e77-4a8c-a863-e97c2a26fee3" containerName="revoke-keys" Nov 01 16:41:36.125278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.125204 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d590df74-4e77-4a8c-a863-e97c2a26fee3" containerName="revoke-keys" Nov 01 16:41:36.125359 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.125308 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="d590df74-4e77-4a8c-a863-e97c2a26fee3" containerName="revoke-keys" Nov 01 16:41:36.128952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.128936 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.134261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.133250 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:41:36.134261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.133304 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:41:36.136384 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.136353 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb"] Nov 01 16:41:36.187998 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.187964 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.188202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.188024 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.188202 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.188048 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqhfs\" (UniqueName: \"kubernetes.io/projected/6dc64f78-4c11-471a-942b-97df6de578ec-kube-api-access-hqhfs\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.289681 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.289643 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.289864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.289705 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.289864 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.289722 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqhfs\" (UniqueName: \"kubernetes.io/projected/6dc64f78-4c11-471a-942b-97df6de578ec-kube-api-access-hqhfs\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.290322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.290301 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.292091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.292061 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.297369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.297343 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqhfs\" (UniqueName: \"kubernetes.io/projected/6dc64f78-4c11-471a-942b-97df6de578ec-kube-api-access-hqhfs\") pod \"maas-api-revoke-sub-e2fc9c5b-4zppb\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.443958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.443885 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:36.719307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:36.719282 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb"] Nov 01 16:41:36.721442 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:41:36.721412 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc64f78_4c11_471a_942b_97df6de578ec.slice/crio-52decca6a026997226271f9d381bdc09f0b353eb379eba98834b1021a7c0087a WatchSource:0}: Error finding container 52decca6a026997226271f9d381bdc09f0b353eb379eba98834b1021a7c0087a: Status 404 returned error can't find the container with id 52decca6a026997226271f9d381bdc09f0b353eb379eba98834b1021a7c0087a Nov 01 16:41:37.436808 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:37.436758 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" event={"ID":"6dc64f78-4c11-471a-942b-97df6de578ec","Type":"ContainerStarted","Data":"85cd21520bebc63a32a546f98a08146c5d942adbb62f398b437f0564ba42ec97"} Nov 01 16:41:37.437221 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:37.436826 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" event={"ID":"6dc64f78-4c11-471a-942b-97df6de578ec","Type":"ContainerStarted","Data":"52decca6a026997226271f9d381bdc09f0b353eb379eba98834b1021a7c0087a"} Nov 01 16:41:37.452085 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:37.452016 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" podStartSLOduration=1.45200314 podStartE2EDuration="1.45200314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:41:37.451098224 +0000 UTC m=+3235.154264161" watchObservedRunningTime="2026-11-01 16:41:37.45200314 +0000 UTC m=+3235.155169108" Nov 01 16:41:38.444225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:38.444189 4573 generic.go:356] "Generic (PLEG): container finished" podID="6dc64f78-4c11-471a-942b-97df6de578ec" containerID="85cd21520bebc63a32a546f98a08146c5d942adbb62f398b437f0564ba42ec97" exitCode=0 Nov 01 16:41:38.444629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:38.444283 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" event={"ID":"6dc64f78-4c11-471a-942b-97df6de578ec","Type":"ContainerDied","Data":"85cd21520bebc63a32a546f98a08146c5d942adbb62f398b437f0564ba42ec97"} Nov 01 16:41:39.578350 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.578323 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:39.622975 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.622949 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-openshift-ca\") pod \"6dc64f78-4c11-471a-942b-97df6de578ec\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " Nov 01 16:41:39.623148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.623045 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dc64f78-4c11-471a-942b-97df6de578ec-kube-api-access-hqhfs\" (UniqueName: \"kubernetes.io/projected/6dc64f78-4c11-471a-942b-97df6de578ec-kube-api-access-hqhfs\") pod \"6dc64f78-4c11-471a-942b-97df6de578ec\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " Nov 01 16:41:39.623148 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.623099 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-xks-ca\") pod \"6dc64f78-4c11-471a-942b-97df6de578ec\" (UID: \"6dc64f78-4c11-471a-942b-97df6de578ec\") " Nov 01 16:41:39.623347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.623322 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-openshift-ca" pod "6dc64f78-4c11-471a-942b-97df6de578ec" (UID: "6dc64f78-4c11-471a-942b-97df6de578ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:41:39.623423 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.623407 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:39.625186 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.625160 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-xks-ca" pod "6dc64f78-4c11-471a-942b-97df6de578ec" (UID: "6dc64f78-4c11-471a-942b-97df6de578ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:41:39.625285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.625224 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc64f78-4c11-471a-942b-97df6de578ec-kube-api-access-hqhfs" pod "6dc64f78-4c11-471a-942b-97df6de578ec" (UID: "6dc64f78-4c11-471a-942b-97df6de578ec"). InnerVolumeSpecName "kube-api-access-hqhfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:41:39.724153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.724072 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqhfs\" (UniqueName: \"kubernetes.io/projected/6dc64f78-4c11-471a-942b-97df6de578ec-kube-api-access-hqhfs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:39.724153 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:39.724102 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dc64f78-4c11-471a-942b-97df6de578ec-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:40.456941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:40.456905 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" Nov 01 16:41:40.456941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:40.456923 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb" event={"ID":"6dc64f78-4c11-471a-942b-97df6de578ec","Type":"ContainerDied","Data":"52decca6a026997226271f9d381bdc09f0b353eb379eba98834b1021a7c0087a"} Nov 01 16:41:40.457170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:40.456970 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52decca6a026997226271f9d381bdc09f0b353eb379eba98834b1021a7c0087a" Nov 01 16:41:46.334104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.334065 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76"] Nov 01 16:41:46.334525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.334510 4573 cpu_manager.go:414] "containerMap: removing container" podUID="6dc64f78-4c11-471a-942b-97df6de578ec" containerName="revoke-keys" Nov 01 16:41:46.334571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.334526 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dc64f78-4c11-471a-942b-97df6de578ec" containerName="revoke-keys" Nov 01 16:41:46.334618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.334610 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dc64f78-4c11-471a-942b-97df6de578ec" containerName="revoke-keys" Nov 01 16:41:46.337934 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.337909 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.345078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.344909 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:41:46.345078 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.345043 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:41:46.350868 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.350837 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76"] Nov 01 16:41:46.386387 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.386354 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65v6l\" (UniqueName: \"kubernetes.io/projected/e3199fff-ff05-46cb-bd9a-981fcfd06599-kube-api-access-65v6l\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.386554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.386525 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.386600 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.386557 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.487874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.487841 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.487874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.487876 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.488110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.487926 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65v6l\" (UniqueName: \"kubernetes.io/projected/e3199fff-ff05-46cb-bd9a-981fcfd06599-kube-api-access-65v6l\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.488606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.488580 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.490247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.490218 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.495307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.495285 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65v6l\" (UniqueName: \"kubernetes.io/projected/e3199fff-ff05-46cb-bd9a-981fcfd06599-kube-api-access-65v6l\") pod \"maas-api-revoke-sub-5a29efb3-c6b76\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.661801 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.661737 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:46.937557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.937524 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76"] Nov 01 16:41:46.939200 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:41:46.939172 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3199fff_ff05_46cb_bd9a_981fcfd06599.slice/crio-d7a8ca7083d40baa1cdb6d986b9d2cdb32f6d759f6d8548e5ae9b7ee84b37c0b WatchSource:0}: Error finding container d7a8ca7083d40baa1cdb6d986b9d2cdb32f6d759f6d8548e5ae9b7ee84b37c0b: Status 404 returned error can't find the container with id d7a8ca7083d40baa1cdb6d986b9d2cdb32f6d759f6d8548e5ae9b7ee84b37c0b Nov 01 16:41:46.978621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:46.978595 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:41:47.500194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:47.500151 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" event={"ID":"e3199fff-ff05-46cb-bd9a-981fcfd06599","Type":"ContainerStarted","Data":"c69ae1bdaa76bad7a9fe18a992718c87df716a92f485f8ceccaa4e8814be62d1"} Nov 01 16:41:47.500709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:47.500222 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" event={"ID":"e3199fff-ff05-46cb-bd9a-981fcfd06599","Type":"ContainerStarted","Data":"d7a8ca7083d40baa1cdb6d986b9d2cdb32f6d759f6d8548e5ae9b7ee84b37c0b"} Nov 01 16:41:47.518417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:47.518350 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" podStartSLOduration=1.518331141 podStartE2EDuration="1.518331141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:41:47.51647016 +0000 UTC m=+3245.219636097" watchObservedRunningTime="2026-11-01 16:41:47.518331141 +0000 UTC m=+3245.221497081" Nov 01 16:41:48.534937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:48.534729 4573 generic.go:356] "Generic (PLEG): container finished" podID="e3199fff-ff05-46cb-bd9a-981fcfd06599" containerID="c69ae1bdaa76bad7a9fe18a992718c87df716a92f485f8ceccaa4e8814be62d1" exitCode=0 Nov 01 16:41:48.534937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:48.534872 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" event={"ID":"e3199fff-ff05-46cb-bd9a-981fcfd06599","Type":"ContainerDied","Data":"c69ae1bdaa76bad7a9fe18a992718c87df716a92f485f8ceccaa4e8814be62d1"} Nov 01 16:41:49.885789 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:49.885536 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:49.941878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:49.941839 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-xks-ca\") pod \"e3199fff-ff05-46cb-bd9a-981fcfd06599\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " Nov 01 16:41:49.942069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:49.941931 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-openshift-ca\") pod \"e3199fff-ff05-46cb-bd9a-981fcfd06599\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " Nov 01 16:41:49.942069 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:49.941970 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3199fff-ff05-46cb-bd9a-981fcfd06599-kube-api-access-65v6l\" (UniqueName: \"kubernetes.io/projected/e3199fff-ff05-46cb-bd9a-981fcfd06599-kube-api-access-65v6l\") pod \"e3199fff-ff05-46cb-bd9a-981fcfd06599\" (UID: \"e3199fff-ff05-46cb-bd9a-981fcfd06599\") " Nov 01 16:41:49.942395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:49.942363 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-openshift-ca" pod "e3199fff-ff05-46cb-bd9a-981fcfd06599" (UID: "e3199fff-ff05-46cb-bd9a-981fcfd06599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:41:49.944052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:49.944026 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-xks-ca" pod "e3199fff-ff05-46cb-bd9a-981fcfd06599" (UID: "e3199fff-ff05-46cb-bd9a-981fcfd06599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:41:49.944524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:49.944509 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3199fff-ff05-46cb-bd9a-981fcfd06599-kube-api-access-65v6l" pod "e3199fff-ff05-46cb-bd9a-981fcfd06599" (UID: "e3199fff-ff05-46cb-bd9a-981fcfd06599"). InnerVolumeSpecName "kube-api-access-65v6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:41:50.043046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.043009 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:50.043046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.043042 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3199fff-ff05-46cb-bd9a-981fcfd06599-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:50.043046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.043053 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65v6l\" (UniqueName: \"kubernetes.io/projected/e3199fff-ff05-46cb-bd9a-981fcfd06599-kube-api-access-65v6l\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:41:50.240073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.239998 4573 scope.go:122] "RemoveContainer" containerID="2a87ee4fbd7207fd9e2c88bd28c1bf7b8a9da19b0a233545b8b25029b602c61e" Nov 01 16:41:50.249111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.249092 4573 scope.go:122] "RemoveContainer" containerID="7799e1e963c26726e0527f805900e537f70768e78bb4075ddce37147faa3363d" Nov 01 16:41:50.257112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.257091 4573 scope.go:122] "RemoveContainer" containerID="b2b223705116180c5c659de96adfe5d320dabf571b28d944aa6386d8dc43328a" Nov 01 16:41:50.265046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.265028 4573 scope.go:122] "RemoveContainer" containerID="2c4f0488d122127dc6ca2e821f6abc2ab40e014883199bdc3fa2313ac9d0aaec" Nov 01 16:41:50.552178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.549171 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" event={"ID":"e3199fff-ff05-46cb-bd9a-981fcfd06599","Type":"ContainerDied","Data":"d7a8ca7083d40baa1cdb6d986b9d2cdb32f6d759f6d8548e5ae9b7ee84b37c0b"} Nov 01 16:41:50.552178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.549215 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76" Nov 01 16:41:50.552178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:50.549233 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7a8ca7083d40baa1cdb6d986b9d2cdb32f6d759f6d8548e5ae9b7ee84b37c0b" Nov 01 16:41:54.363832 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:41:54.363733 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:41:57.189529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:41:57.189491 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:42:01.835128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.835086 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn"] Nov 01 16:42:01.839804 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.839766 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-2h2vn"] Nov 01 16:42:01.855263 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.855230 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq"] Nov 01 16:42:01.855701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.855687 4573 cpu_manager.go:414] "containerMap: removing container" podUID="e3199fff-ff05-46cb-bd9a-981fcfd06599" containerName="revoke-keys" Nov 01 16:42:01.855774 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.855702 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3199fff-ff05-46cb-bd9a-981fcfd06599" containerName="revoke-keys" Nov 01 16:42:01.855828 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.855801 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3199fff-ff05-46cb-bd9a-981fcfd06599" containerName="revoke-keys" Nov 01 16:42:01.860871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.860850 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:01.865140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.865097 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:42:01.865542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.865517 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:42:01.867081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.867054 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq"] Nov 01 16:42:01.955070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.955033 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:01.955248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.955089 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:01.955248 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:01.955224 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jshzt\" (UniqueName: \"kubernetes.io/projected/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-kube-api-access-jshzt\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:02.056476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.056437 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:02.056660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.056501 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jshzt\" (UniqueName: \"kubernetes.io/projected/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-kube-api-access-jshzt\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:02.056660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.056572 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:02.057447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.057422 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:02.058873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.058850 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:02.064329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.064308 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jshzt\" (UniqueName: \"kubernetes.io/projected/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-kube-api-access-jshzt\") pod \"maas-api-revoke-sub-3d4156f1-d8cpq\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:02.178375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.178338 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:02.454499 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:42:02.454463 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d953e71_856a_46ff_9feb_88a7dcaf3bbd.slice/crio-71f6cfbb5ed8b13d99c282e058874c9191906ef68f9b6fcf8fa30421472c7dc6 WatchSource:0}: Error finding container 71f6cfbb5ed8b13d99c282e058874c9191906ef68f9b6fcf8fa30421472c7dc6: Status 404 returned error can't find the container with id 71f6cfbb5ed8b13d99c282e058874c9191906ef68f9b6fcf8fa30421472c7dc6 Nov 01 16:42:02.456999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.456973 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d590df74-4e77-4a8c-a863-e97c2a26fee3" path="/var/lib/kubelet/pods/d590df74-4e77-4a8c-a863-e97c2a26fee3/volumes" Nov 01 16:42:02.457421 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.457331 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq"] Nov 01 16:42:02.618230 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:02.618193 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" event={"ID":"3d953e71-856a-46ff-9feb-88a7dcaf3bbd","Type":"ContainerStarted","Data":"71f6cfbb5ed8b13d99c282e058874c9191906ef68f9b6fcf8fa30421472c7dc6"} Nov 01 16:42:03.625214 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:03.625180 4573 generic.go:356] "Generic (PLEG): container finished" podID="3d953e71-856a-46ff-9feb-88a7dcaf3bbd" containerID="afa65aeffde05c699142c3091408fa79b90978ba42890844e6ce76a2032b1bf9" exitCode=0 Nov 01 16:42:03.625635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:03.625222 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" event={"ID":"3d953e71-856a-46ff-9feb-88a7dcaf3bbd","Type":"ContainerDied","Data":"afa65aeffde05c699142c3091408fa79b90978ba42890844e6ce76a2032b1bf9"} Nov 01 16:42:04.767317 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.767290 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:04.885580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.885494 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-openshift-ca\") pod \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " Nov 01 16:42:04.885580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.885531 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-kube-api-access-jshzt\" (UniqueName: \"kubernetes.io/projected/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-kube-api-access-jshzt\") pod \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " Nov 01 16:42:04.885580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.885568 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-xks-ca\") pod \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\" (UID: \"3d953e71-856a-46ff-9feb-88a7dcaf3bbd\") " Nov 01 16:42:04.885997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.885966 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-openshift-ca" pod "3d953e71-856a-46ff-9feb-88a7dcaf3bbd" (UID: "3d953e71-856a-46ff-9feb-88a7dcaf3bbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:42:04.887711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.887685 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-xks-ca" pod "3d953e71-856a-46ff-9feb-88a7dcaf3bbd" (UID: "3d953e71-856a-46ff-9feb-88a7dcaf3bbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:42:04.887839 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.887714 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-kube-api-access-jshzt" pod "3d953e71-856a-46ff-9feb-88a7dcaf3bbd" (UID: "3d953e71-856a-46ff-9feb-88a7dcaf3bbd"). InnerVolumeSpecName "kube-api-access-jshzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:42:04.986574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.986542 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:42:04.986574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.986570 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jshzt\" (UniqueName: \"kubernetes.io/projected/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-kube-api-access-jshzt\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:42:04.986574 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:04.986580 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d953e71-856a-46ff-9feb-88a7dcaf3bbd-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:42:05.637893 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:05.637865 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" Nov 01 16:42:05.638090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:05.637901 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq" event={"ID":"3d953e71-856a-46ff-9feb-88a7dcaf3bbd","Type":"ContainerDied","Data":"71f6cfbb5ed8b13d99c282e058874c9191906ef68f9b6fcf8fa30421472c7dc6"} Nov 01 16:42:05.638090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:05.637950 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f6cfbb5ed8b13d99c282e058874c9191906ef68f9b6fcf8fa30421472c7dc6" Nov 01 16:42:12.648151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:12.648113 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:42:16.775673 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:16.775638 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:42:20.017312 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:20.017271 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:20.448225 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:20.448184 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:20.548143 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:20.548115 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:23.875197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:23.875108 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:23.974687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:23.974654 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:24.363639 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:42:24.363604 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:42:25.073650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.073613 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:25.089037 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.089004 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz"] Nov 01 16:42:25.089492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.089476 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3d953e71-856a-46ff-9feb-88a7dcaf3bbd" containerName="revoke-keys" Nov 01 16:42:25.089601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.089493 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d953e71-856a-46ff-9feb-88a7dcaf3bbd" containerName="revoke-keys" Nov 01 16:42:25.089601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.089569 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d953e71-856a-46ff-9feb-88a7dcaf3bbd" containerName="revoke-keys" Nov 01 16:42:25.092976 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.092953 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.107996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.107967 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz"] Nov 01 16:42:25.111276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.111251 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ee2b3d-serving-cert\"" Nov 01 16:42:25.111458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.111359 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 16:42:25.111458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.111364 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 16:42:25.129942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.129912 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mmwq\" (UniqueName: \"kubernetes.io/projected/355abddc-fca2-4ba6-ad5e-993985cbb92b-kube-api-access-4mmwq\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.130142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.130006 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.130252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.130130 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.130332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.130285 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355abddc-fca2-4ba6-ad5e-993985cbb92b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.231608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.231571 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.231830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.231637 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.231830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.231703 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355abddc-fca2-4ba6-ad5e-993985cbb92b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.231830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.231739 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mmwq\" (UniqueName: \"kubernetes.io/projected/355abddc-fca2-4ba6-ad5e-993985cbb92b-kube-api-access-4mmwq\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.232469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.232444 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355abddc-fca2-4ba6-ad5e-993985cbb92b-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.234128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.234107 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.234236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.234175 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.244615 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.244588 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mmwq\" (UniqueName: \"kubernetes.io/projected/355abddc-fca2-4ba6-ad5e-993985cbb92b-kube-api-access-4mmwq\") pod \"maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.304482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.304444 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:25.407951 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.407919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:25.412925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.412890 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:25.691849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.691816 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz"] Nov 01 16:42:25.693908 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:42:25.693873 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355abddc_fca2_4ba6_ad5e_993985cbb92b.slice/crio-fd664ec5a6acdb6d6a4748c0b64b5d9485f0f4e0a69b88dde8ab738a3f08c1c5 WatchSource:0}: Error finding container fd664ec5a6acdb6d6a4748c0b64b5d9485f0f4e0a69b88dde8ab738a3f08c1c5: Status 404 returned error can't find the container with id fd664ec5a6acdb6d6a4748c0b64b5d9485f0f4e0a69b88dde8ab738a3f08c1c5 Nov 01 16:42:25.695152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.695133 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:42:25.765453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:25.765415 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" event={"ID":"355abddc-fca2-4ba6-ad5e-993985cbb92b","Type":"ContainerStarted","Data":"fd664ec5a6acdb6d6a4748c0b64b5d9485f0f4e0a69b88dde8ab738a3f08c1c5"} Nov 01 16:42:26.043297 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:26.043216 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:26.773170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:26.773135 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" event={"ID":"355abddc-fca2-4ba6-ad5e-993985cbb92b","Type":"ContainerStarted","Data":"967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a"} Nov 01 16:42:26.773625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:26.773317 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:26.786372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:26.786346 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:26.789467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:26.789422 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" podStartSLOduration=1.203109809 podStartE2EDuration="1.789391034s" totalImagesPullingTime="586.281225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:42:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:42:26.788579956 +0000 UTC m=+3284.491745893" watchObservedRunningTime="2026-11-01 16:42:26.789391034 +0000 UTC m=+3284.492556970" Nov 01 16:42:27.254817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:27.254733 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:42:27.506813 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:27.506722 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:28.182546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:28.182513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:28.786305 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:28.786266 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:29.486161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:29.486129 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:30.083036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:30.083008 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:30.683744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:30.683703 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:31.280764 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:31.280723 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:31.956089 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:31.956046 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:32.690211 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:32.690178 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:32.783837 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:32.783811 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:42:32.870543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:32.870492 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:33.358494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:33.358461 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:34.065966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:34.065931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:34.802677 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:34.802648 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:35.393219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:35.393181 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:35.983937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:35.983904 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:36.578947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:36.578909 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:37.182059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:37.182018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:37.882416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:37.882381 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:38.495004 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:38.494968 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:38.872016 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:38.871984 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:39.094026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:39.093995 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:39.468152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:39.468121 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:39.566349 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:39.566317 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:39.669937 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:39.669888 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:39.818963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:39.818880 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:40.593286 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:40.593241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:41.346495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:41.346463 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:41.601040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:41.600958 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:42.196132 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:42.196101 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:42.887719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:42.887678 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:43.189708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:43.189626 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:42:43.441554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:43.441483 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:42:43.453510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:43.453476 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:42:43.453957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:43.453934 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:42:43.465941 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:43.465917 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:42:43.711728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:43.711643 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:44.581415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:44.581382 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:45.297433 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:45.297397 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:45.975073 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:45.975037 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:46.701549 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:46.701518 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:47.396206 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:47.396172 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:47.994695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:47.994660 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:48.594924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:48.594895 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:49.198427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:49.198392 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:49.788880 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:49.788844 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:50.297531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:50.297503 4573 scope.go:122] "RemoveContainer" containerID="8c289f41892a8ff9d249e8fbf1941e54f54b91ceee15bf08c1f2ad99d6a8a1a8" Nov 01 16:42:50.417588 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:50.417548 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:51.088862 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:51.088827 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:51.692611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:51.692580 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:52.289285 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:52.289249 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:53.544533 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:53.544495 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:53.645688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:53.645652 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:54.301921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:54.301861 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:54.366515 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:42:54.366475 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:42:55.012660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:55.012624 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:55.722473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:55.722436 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:56.309420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:56.309387 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:56.991853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:56.991821 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:57.605662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:57.605626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:58.324227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:58.324196 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:58.687040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:58.687006 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:42:59.274830 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:59.274788 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:42:59.985057 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:42:59.985024 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:00.596672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:00.596643 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:01.300112 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:01.300075 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:01.999713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:01.999683 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:02.710874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:02.710835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:03.360398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:03.360365 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:03.991953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:03.991919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:04.598743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:04.598709 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:05.277343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:05.277311 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:05.993662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:05.993626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:06.609055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:06.608998 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:07.201738 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:07.201704 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:07.904461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:07.904428 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:08.494723 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:08.494691 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:09.099427 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:09.099390 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:09.699657 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:09.699621 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:10.371921 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:10.371889 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:11.166500 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:11.166470 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:11.920383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:11.920338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:12.706431 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:12.706393 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:13.282959 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:13.282924 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:13.525544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:13.525512 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:14.178253 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:14.178212 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:14.792020 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:14.791982 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:15.516895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:15.516856 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:16.197396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:16.197361 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:16.999694 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:16.999660 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:17.298174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:17.298090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:17.399566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:17.399513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:17.655049 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:17.655015 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:18.298232 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:18.298196 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:18.905948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:18.905890 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:19.695593 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:19.695555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:20.296026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:20.295988 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:20.901010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:20.900969 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:21.021991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:21.021953 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz"] Nov 01 16:43:21.025966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:21.025941 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-f64kz"] Nov 01 16:43:21.692244 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:21.692209 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:21.945881 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:21.945791 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:22.456381 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:22.456340 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdddb1db-0469-4478-b288-badbc23d710f" path="/var/lib/kubelet/pods/bdddb1db-0469-4478-b288-badbc23d710f/volumes" Nov 01 16:43:22.683087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:22.683049 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:43:22.801931 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:22.801828 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:23.586742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:23.586707 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:24.196653 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:24.196617 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:24.363087 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:43:24.363051 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:43:24.886689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:24.886656 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:25.722336 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:25.722301 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:26.398472 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:26.398430 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:26.717058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:26.716976 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:26.995530 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:26.995448 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:27.605033 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:27.605002 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:28.328552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:28.328514 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:29.010846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:29.010810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:29.705392 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:29.705360 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:30.306674 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:30.306639 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:31.106671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:31.106640 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:31.900034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:31.900002 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:32.162495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.162418 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:32.253524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.253493 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:32.544560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.544481 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz"] Nov 01 16:43:32.544878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.544769 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" podUID="355abddc-fca2-4ba6-ad5e-993985cbb92b" containerName="maas-api" containerID="cri-o://967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a" gracePeriod=30 Nov 01 16:43:32.589800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.589762 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:32.738233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.738198 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:32.779152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.779106 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" podUID="355abddc-fca2-4ba6-ad5e-993985cbb92b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.90:8443/health\": dial tcp 10.133.1.90:8443: connect: connection refused" Nov 01 16:43:32.855479 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.855440 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:32.996546 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.996519 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:43:32.998663 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:32.998642 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:33.001110 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.001092 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/355abddc-fca2-4ba6-ad5e-993985cbb92b-kube-api-access-4mmwq\" (UniqueName: \"kubernetes.io/projected/355abddc-fca2-4ba6-ad5e-993985cbb92b-kube-api-access-4mmwq\") pod \"355abddc-fca2-4ba6-ad5e-993985cbb92b\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " Nov 01 16:43:33.001191 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.001123 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-metrics-certs\") pod \"355abddc-fca2-4ba6-ad5e-993985cbb92b\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " Nov 01 16:43:33.001252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.001209 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/355abddc-fca2-4ba6-ad5e-993985cbb92b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355abddc-fca2-4ba6-ad5e-993985cbb92b-openshift-service-ca\") pod \"355abddc-fca2-4ba6-ad5e-993985cbb92b\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " Nov 01 16:43:33.001318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.001267 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-maas-api-tls\") pod \"355abddc-fca2-4ba6-ad5e-993985cbb92b\" (UID: \"355abddc-fca2-4ba6-ad5e-993985cbb92b\") " Nov 01 16:43:33.001558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.001531 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/355abddc-fca2-4ba6-ad5e-993985cbb92b-openshift-service-ca" pod "355abddc-fca2-4ba6-ad5e-993985cbb92b" (UID: "355abddc-fca2-4ba6-ad5e-993985cbb92b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:43:33.003308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.003275 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-metrics-certs" pod "355abddc-fca2-4ba6-ad5e-993985cbb92b" (UID: "355abddc-fca2-4ba6-ad5e-993985cbb92b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:43:33.003412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.003311 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-maas-api-tls" pod "355abddc-fca2-4ba6-ad5e-993985cbb92b" (UID: "355abddc-fca2-4ba6-ad5e-993985cbb92b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:43:33.003412 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.003375 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/355abddc-fca2-4ba6-ad5e-993985cbb92b-kube-api-access-4mmwq" pod "355abddc-fca2-4ba6-ad5e-993985cbb92b" (UID: "355abddc-fca2-4ba6-ad5e-993985cbb92b"). InnerVolumeSpecName "kube-api-access-4mmwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:33.102096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.102016 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:33.102096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.102046 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mmwq\" (UniqueName: \"kubernetes.io/projected/355abddc-fca2-4ba6-ad5e-993985cbb92b-kube-api-access-4mmwq\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:33.102096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.102058 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/355abddc-fca2-4ba6-ad5e-993985cbb92b-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:33.102096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.102067 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/355abddc-fca2-4ba6-ad5e-993985cbb92b-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:33.134734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.134704 4573 generic.go:356] "Generic (PLEG): container finished" podID="355abddc-fca2-4ba6-ad5e-993985cbb92b" containerID="967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a" exitCode=0 Nov 01 16:43:33.134912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.134745 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" Nov 01 16:43:33.134912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.134788 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" event={"ID":"355abddc-fca2-4ba6-ad5e-993985cbb92b","Type":"ContainerDied","Data":"967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a"} Nov 01 16:43:33.134912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.134840 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz" event={"ID":"355abddc-fca2-4ba6-ad5e-993985cbb92b","Type":"ContainerDied","Data":"fd664ec5a6acdb6d6a4748c0b64b5d9485f0f4e0a69b88dde8ab738a3f08c1c5"} Nov 01 16:43:33.134912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.134864 4573 scope.go:122] "RemoveContainer" containerID="967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a" Nov 01 16:43:33.144984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.144968 4573 scope.go:122] "RemoveContainer" containerID="967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a" Nov 01 16:43:33.145236 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:43:33.145218 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a\": container with ID starting with 967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a not found: ID does not exist" containerID="967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a" Nov 01 16:43:33.145290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.145245 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a"} err="failed to get container status \"967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a\": rpc error: code = NotFound desc = could not find container \"967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a\": container with ID starting with 967e899dcd5b9e6cf3b3b778e5c2565a2fb022be4fbf9d2000da74ffe153b62a not found: ID does not exist" Nov 01 16:43:33.156138 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.156115 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz"] Nov 01 16:43:33.164707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.164681 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee2b3d-767fcb56ff-7l7cz"] Nov 01 16:43:33.340971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:33.340931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:34.462050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:34.462015 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="355abddc-fca2-4ba6-ad5e-993985cbb92b" path="/var/lib/kubelet/pods/355abddc-fca2-4ba6-ad5e-993985cbb92b/volumes" Nov 01 16:43:34.642475 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:34.642442 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:43:39.139146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:39.139107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:39.227009 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:39.226946 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:39.465973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:39.465886 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:43:40.022596 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.022553 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl"] Nov 01 16:43:40.026792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.026760 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-9dkvl"] Nov 01 16:43:40.455904 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.455870 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e647bfc3-44ac-4aaa-84ee-7e45701194b2" path="/var/lib/kubelet/pods/e647bfc3-44ac-4aaa-84ee-7e45701194b2/volumes" Nov 01 16:43:40.659610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.659575 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq"] Nov 01 16:43:40.666909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.666880 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d8cpq"] Nov 01 16:43:40.685665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.685634 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2"] Nov 01 16:43:40.686120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.686101 4573 cpu_manager.go:414] "containerMap: removing container" podUID="355abddc-fca2-4ba6-ad5e-993985cbb92b" containerName="maas-api" Nov 01 16:43:40.686209 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.686122 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="355abddc-fca2-4ba6-ad5e-993985cbb92b" containerName="maas-api" Nov 01 16:43:40.686272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.686254 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="355abddc-fca2-4ba6-ad5e-993985cbb92b" containerName="maas-api" Nov 01 16:43:40.690815 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.690797 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.694985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.694962 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:43:40.695284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.695166 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:43:40.699139 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.699117 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2"] Nov 01 16:43:40.779137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.779057 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.779292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.779192 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.779292 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.779234 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7nnx\" (UniqueName: \"kubernetes.io/projected/21832286-b3c2-4059-b9ea-7cb04e87db22-kube-api-access-c7nnx\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.880599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.880559 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.880823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.880622 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7nnx\" (UniqueName: \"kubernetes.io/projected/21832286-b3c2-4059-b9ea-7cb04e87db22-kube-api-access-c7nnx\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.880823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.880678 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.881422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.881400 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.882964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.882947 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:40.889401 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:40.889382 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7nnx\" (UniqueName: \"kubernetes.io/projected/21832286-b3c2-4059-b9ea-7cb04e87db22-kube-api-access-c7nnx\") pod \"maas-api-revoke-sub-3d4156f1-p8cm2\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:41.005348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:41.005315 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:41.280608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:41.280582 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2"] Nov 01 16:43:41.282786 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:43:41.282742 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21832286_b3c2_4059_b9ea_7cb04e87db22.slice/crio-9743ee2cd70cdd54d38b90082ad2aba2c85d3c23cf6da07c270a344dbb0c8675 WatchSource:0}: Error finding container 9743ee2cd70cdd54d38b90082ad2aba2c85d3c23cf6da07c270a344dbb0c8675: Status 404 returned error can't find the container with id 9743ee2cd70cdd54d38b90082ad2aba2c85d3c23cf6da07c270a344dbb0c8675 Nov 01 16:43:42.187570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:42.187537 4573 generic.go:356] "Generic (PLEG): container finished" podID="21832286-b3c2-4059-b9ea-7cb04e87db22" containerID="012aef113fb884055f1f2ebe35b1b999030a200fa43d195effbab9f48c870a6c" exitCode=0 Nov 01 16:43:42.187968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:42.187617 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" event={"ID":"21832286-b3c2-4059-b9ea-7cb04e87db22","Type":"ContainerDied","Data":"012aef113fb884055f1f2ebe35b1b999030a200fa43d195effbab9f48c870a6c"} Nov 01 16:43:42.187968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:42.187669 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" event={"ID":"21832286-b3c2-4059-b9ea-7cb04e87db22","Type":"ContainerStarted","Data":"9743ee2cd70cdd54d38b90082ad2aba2c85d3c23cf6da07c270a344dbb0c8675"} Nov 01 16:43:42.457710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:42.457681 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d953e71-856a-46ff-9feb-88a7dcaf3bbd" path="/var/lib/kubelet/pods/3d953e71-856a-46ff-9feb-88a7dcaf3bbd/volumes" Nov 01 16:43:43.329845 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.329818 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:43.405062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.405023 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-xks-ca\") pod \"21832286-b3c2-4059-b9ea-7cb04e87db22\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " Nov 01 16:43:43.405264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.405069 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-openshift-ca\") pod \"21832286-b3c2-4059-b9ea-7cb04e87db22\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " Nov 01 16:43:43.405264 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.405168 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21832286-b3c2-4059-b9ea-7cb04e87db22-kube-api-access-c7nnx\" (UniqueName: \"kubernetes.io/projected/21832286-b3c2-4059-b9ea-7cb04e87db22-kube-api-access-c7nnx\") pod \"21832286-b3c2-4059-b9ea-7cb04e87db22\" (UID: \"21832286-b3c2-4059-b9ea-7cb04e87db22\") " Nov 01 16:43:43.405465 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.405441 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-openshift-ca" pod "21832286-b3c2-4059-b9ea-7cb04e87db22" (UID: "21832286-b3c2-4059-b9ea-7cb04e87db22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:43:43.407313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.407287 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-xks-ca" pod "21832286-b3c2-4059-b9ea-7cb04e87db22" (UID: "21832286-b3c2-4059-b9ea-7cb04e87db22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:43:43.407313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.407287 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21832286-b3c2-4059-b9ea-7cb04e87db22-kube-api-access-c7nnx" pod "21832286-b3c2-4059-b9ea-7cb04e87db22" (UID: "21832286-b3c2-4059-b9ea-7cb04e87db22"). InnerVolumeSpecName "kube-api-access-c7nnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:43:43.506289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.506200 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:43.506289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.506229 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21832286-b3c2-4059-b9ea-7cb04e87db22-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:43.506289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:43.506240 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7nnx\" (UniqueName: \"kubernetes.io/projected/21832286-b3c2-4059-b9ea-7cb04e87db22-kube-api-access-c7nnx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:43:44.199879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:44.199843 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" Nov 01 16:43:44.199879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:44.199856 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2" event={"ID":"21832286-b3c2-4059-b9ea-7cb04e87db22","Type":"ContainerDied","Data":"9743ee2cd70cdd54d38b90082ad2aba2c85d3c23cf6da07c270a344dbb0c8675"} Nov 01 16:43:44.200114 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:44.199908 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9743ee2cd70cdd54d38b90082ad2aba2c85d3c23cf6da07c270a344dbb0c8675" Nov 01 16:43:50.022159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:50.022116 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr"] Nov 01 16:43:50.024318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:50.024292 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-658zr"] Nov 01 16:43:50.455743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:50.455713 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c157e8a-6c9e-4aa9-a81c-7015cac4801a" path="/var/lib/kubelet/pods/1c157e8a-6c9e-4aa9-a81c-7015cac4801a/volumes" Nov 01 16:43:52.344792 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:52.344732 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:43:54.377820 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:43:54.370425 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:43:56.880106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:43:56.880058 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:44:02.826307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:02.826276 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:44:04.343655 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:04.343624 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:44:09.869068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:09.869033 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:44:17.022567 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:17.022525 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g"] Nov 01 16:44:17.025611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:17.025578 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-9n67g"] Nov 01 16:44:18.456531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:18.456496 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91faa95d-32b4-42a4-8ef9-b5d35762f4f4" path="/var/lib/kubelet/pods/91faa95d-32b4-42a4-8ef9-b5d35762f4f4/volumes" Nov 01 16:44:24.363169 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:44:24.363132 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:44:27.038289 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:27.038249 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm"] Nov 01 16:44:27.041999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:27.041965 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rs4qm"] Nov 01 16:44:28.455158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:28.455118 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e6beffd-779a-4c53-a78d-a5beda2ba056" path="/var/lib/kubelet/pods/6e6beffd-779a-4c53-a78d-a5beda2ba056/volumes" Nov 01 16:44:32.997573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:32.997535 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:44:33.107866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:33.107834 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:44:47.022389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:47.022350 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t"] Nov 01 16:44:47.024411 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:47.024387 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2pl6t"] Nov 01 16:44:48.456478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:48.456434 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffde061e-8126-4ba0-b8fb-0f2dc51b7411" path="/var/lib/kubelet/pods/ffde061e-8126-4ba0-b8fb-0f2dc51b7411/volumes" Nov 01 16:44:50.336714 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:50.336683 4573 scope.go:122] "RemoveContainer" containerID="6531bb78e3c62617c43c3dbe29a8495a310e51fbdd204ab28cd90a27e408f538" Nov 01 16:44:50.345379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:50.345357 4573 scope.go:122] "RemoveContainer" containerID="f0d9c63dcc3126253b753d197145a4952a9d70048f1cf54b914c32979b35fc16" Nov 01 16:44:50.353274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:50.353256 4573 scope.go:122] "RemoveContainer" containerID="4c24b91f865f9f6533c69afd966b11fce2aa741a06c30e8292b33ce65a5ae135" Nov 01 16:44:53.584222 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:53.584189 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:44:53.656207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:44:53.656172 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:44:54.363977 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:44:54.363938 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:45:00.139722 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.139679 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5"] Nov 01 16:45:00.140197 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.140179 4573 cpu_manager.go:414] "containerMap: removing container" podUID="21832286-b3c2-4059-b9ea-7cb04e87db22" containerName="revoke-keys" Nov 01 16:45:00.140247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.140198 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21832286-b3c2-4059-b9ea-7cb04e87db22" containerName="revoke-keys" Nov 01 16:45:00.140315 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.140306 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="21832286-b3c2-4059-b9ea-7cb04e87db22" containerName="revoke-keys" Nov 01 16:45:00.145272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.145254 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:00.153806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.153771 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5"] Nov 01 16:45:00.163283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.163252 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:45:00.269389 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.269345 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nb5m\" (UniqueName: \"kubernetes.io/projected/ff45c06b-5eb5-42d6-9793-dabf9f642465-kube-api-access-9nb5m\") pod \"maas-api-key-cleanup-29892525-7zqx5\" (UID: \"ff45c06b-5eb5-42d6-9793-dabf9f642465\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:00.269581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.269494 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff45c06b-5eb5-42d6-9793-dabf9f642465-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892525-7zqx5\" (UID: \"ff45c06b-5eb5-42d6-9793-dabf9f642465\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:00.370374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.370332 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff45c06b-5eb5-42d6-9793-dabf9f642465-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892525-7zqx5\" (UID: \"ff45c06b-5eb5-42d6-9793-dabf9f642465\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:00.370578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.370416 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nb5m\" (UniqueName: \"kubernetes.io/projected/ff45c06b-5eb5-42d6-9793-dabf9f642465-kube-api-access-9nb5m\") pod \"maas-api-key-cleanup-29892525-7zqx5\" (UID: \"ff45c06b-5eb5-42d6-9793-dabf9f642465\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:00.371044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.371022 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff45c06b-5eb5-42d6-9793-dabf9f642465-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892525-7zqx5\" (UID: \"ff45c06b-5eb5-42d6-9793-dabf9f642465\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:00.379724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.379703 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nb5m\" (UniqueName: \"kubernetes.io/projected/ff45c06b-5eb5-42d6-9793-dabf9f642465-kube-api-access-9nb5m\") pod \"maas-api-key-cleanup-29892525-7zqx5\" (UID: \"ff45c06b-5eb5-42d6-9793-dabf9f642465\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:00.465238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.465164 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:00.739170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:00.739133 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5"] Nov 01 16:45:00.740682 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:45:00.740653 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff45c06b_5eb5_42d6_9793_dabf9f642465.slice/crio-574caade7892a24c8aa72fe55b9894cc6b48e1c15a1ae8d74792a9fff8506373 WatchSource:0}: Error finding container 574caade7892a24c8aa72fe55b9894cc6b48e1c15a1ae8d74792a9fff8506373: Status 404 returned error can't find the container with id 574caade7892a24c8aa72fe55b9894cc6b48e1c15a1ae8d74792a9fff8506373 Nov 01 16:45:01.620345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:01.620310 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerID="89a73229b7a173ef6f6ce00c95018e393e62e2af706c8a48ab9f3da6838813bb" exitCode=60 Nov 01 16:45:01.620797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:01.620354 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" event={"ID":"ff45c06b-5eb5-42d6-9793-dabf9f642465","Type":"ContainerDied","Data":"89a73229b7a173ef6f6ce00c95018e393e62e2af706c8a48ab9f3da6838813bb"} Nov 01 16:45:01.620797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:01.620397 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" event={"ID":"ff45c06b-5eb5-42d6-9793-dabf9f642465","Type":"ContainerStarted","Data":"574caade7892a24c8aa72fe55b9894cc6b48e1c15a1ae8d74792a9fff8506373"} Nov 01 16:45:01.620797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:01.620642 4573 scope.go:122] "RemoveContainer" containerID="89a73229b7a173ef6f6ce00c95018e393e62e2af706c8a48ab9f3da6838813bb" Nov 01 16:45:02.627247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:02.627204 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerID="a5dabaf5b067953b3f86aaf4d7e5d435443b6769baf1890ad39581423defe5d7" exitCode=60 Nov 01 16:45:02.627796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:02.627293 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" event={"ID":"ff45c06b-5eb5-42d6-9793-dabf9f642465","Type":"ContainerDied","Data":"a5dabaf5b067953b3f86aaf4d7e5d435443b6769baf1890ad39581423defe5d7"} Nov 01 16:45:02.627796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:02.627371 4573 scope.go:122] "RemoveContainer" containerID="89a73229b7a173ef6f6ce00c95018e393e62e2af706c8a48ab9f3da6838813bb" Nov 01 16:45:02.627796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:02.627702 4573 scope.go:122] "RemoveContainer" containerID="a5dabaf5b067953b3f86aaf4d7e5d435443b6769baf1890ad39581423defe5d7" Nov 01 16:45:02.628035 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:45:02.628013 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892525-7zqx5_odh-ai-gateway-infra(ff45c06b-5eb5-42d6-9793-dabf9f642465)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" Nov 01 16:45:03.635062 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:03.635034 4573 scope.go:122] "RemoveContainer" containerID="a5dabaf5b067953b3f86aaf4d7e5d435443b6769baf1890ad39581423defe5d7" Nov 01 16:45:03.635535 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:45:03.635351 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892525-7zqx5_odh-ai-gateway-infra(ff45c06b-5eb5-42d6-9793-dabf9f642465)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" Nov 01 16:45:05.023992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:05.023950 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s"] Nov 01 16:45:05.026439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:05.026411 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tfz4s"] Nov 01 16:45:06.457364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:06.457325 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb715982-66ce-4ea8-8d53-dcfe3bf487db" path="/var/lib/kubelet/pods/eb715982-66ce-4ea8-8d53-dcfe3bf487db/volumes" Nov 01 16:45:14.450354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:14.450313 4573 scope.go:122] "RemoveContainer" containerID="a5dabaf5b067953b3f86aaf4d7e5d435443b6769baf1890ad39581423defe5d7" Nov 01 16:45:15.702798 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:15.702689 4573 generic.go:356] "Generic (PLEG): container finished" podID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerID="88a5e6326de9dda3bf90b579fd231dd29a711369d953d67cbf5f3283f552f3d2" exitCode=60 Nov 01 16:45:15.703274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:15.702786 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" event={"ID":"ff45c06b-5eb5-42d6-9793-dabf9f642465","Type":"ContainerDied","Data":"88a5e6326de9dda3bf90b579fd231dd29a711369d953d67cbf5f3283f552f3d2"} Nov 01 16:45:15.703274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:15.702860 4573 scope.go:122] "RemoveContainer" containerID="a5dabaf5b067953b3f86aaf4d7e5d435443b6769baf1890ad39581423defe5d7" Nov 01 16:45:15.703274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:15.703188 4573 scope.go:122] "RemoveContainer" containerID="88a5e6326de9dda3bf90b579fd231dd29a711369d953d67cbf5f3283f552f3d2" Nov 01 16:45:15.703491 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:45:15.703468 4573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892525-7zqx5_odh-ai-gateway-infra(ff45c06b-5eb5-42d6-9793-dabf9f642465)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" Nov 01 16:45:16.728522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:16.728493 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5"] Nov 01 16:45:16.869708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:16.869683 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:17.024807 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.024711 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff45c06b-5eb5-42d6-9793-dabf9f642465-kube-api-access-9nb5m\" (UniqueName: \"kubernetes.io/projected/ff45c06b-5eb5-42d6-9793-dabf9f642465-kube-api-access-9nb5m\") pod \"ff45c06b-5eb5-42d6-9793-dabf9f642465\" (UID: \"ff45c06b-5eb5-42d6-9793-dabf9f642465\") " Nov 01 16:45:17.024963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.024811 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff45c06b-5eb5-42d6-9793-dabf9f642465-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff45c06b-5eb5-42d6-9793-dabf9f642465-maas-api-service-ca\") pod \"ff45c06b-5eb5-42d6-9793-dabf9f642465\" (UID: \"ff45c06b-5eb5-42d6-9793-dabf9f642465\") " Nov 01 16:45:17.025174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.025153 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff45c06b-5eb5-42d6-9793-dabf9f642465-maas-api-service-ca" pod "ff45c06b-5eb5-42d6-9793-dabf9f642465" (UID: "ff45c06b-5eb5-42d6-9793-dabf9f642465"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:45:17.026742 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.026716 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff45c06b-5eb5-42d6-9793-dabf9f642465-kube-api-access-9nb5m" pod "ff45c06b-5eb5-42d6-9793-dabf9f642465" (UID: "ff45c06b-5eb5-42d6-9793-dabf9f642465"). InnerVolumeSpecName "kube-api-access-9nb5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:45:17.125823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.125790 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nb5m\" (UniqueName: \"kubernetes.io/projected/ff45c06b-5eb5-42d6-9793-dabf9f642465-kube-api-access-9nb5m\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:45:17.125823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.125820 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff45c06b-5eb5-42d6-9793-dabf9f642465-maas-api-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:45:17.715949 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.715918 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" Nov 01 16:45:17.716160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.715955 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5" event={"ID":"ff45c06b-5eb5-42d6-9793-dabf9f642465","Type":"ContainerDied","Data":"574caade7892a24c8aa72fe55b9894cc6b48e1c15a1ae8d74792a9fff8506373"} Nov 01 16:45:17.716160 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.716024 4573 scope.go:122] "RemoveContainer" containerID="88a5e6326de9dda3bf90b579fd231dd29a711369d953d67cbf5f3283f552f3d2" Nov 01 16:45:17.748034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.748011 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5"] Nov 01 16:45:17.751895 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:17.751873 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892525-7zqx5"] Nov 01 16:45:18.459345 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:18.459310 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" path="/var/lib/kubelet/pods/ff45c06b-5eb5-42d6-9793-dabf9f642465/volumes" Nov 01 16:45:19.984396 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:19.984363 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:45:20.081884 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:20.081843 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:45:21.081355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:21.081317 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:45:24.362691 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:45:24.362610 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:45:25.023103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.023066 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7"] Nov 01 16:45:25.025018 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.024994 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-ln7g7"] Nov 01 16:45:25.696901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.696854 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s"] Nov 01 16:45:25.697309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.697293 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:25.697309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.697305 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:25.697388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.697318 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:25.697388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.697323 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:25.697388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.697341 4573 cpu_manager.go:414] "containerMap: removing container" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:25.697388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.697346 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:25.697525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.697406 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:25.697525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.697415 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:25.701854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.701830 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.705671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.705633 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:45:25.705823 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.705730 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:45:25.708215 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.708194 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s"] Nov 01 16:45:25.826650 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.826609 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.827053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.827028 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nfht\" (UniqueName: \"kubernetes.io/projected/a29ea145-1600-4f8c-b46e-a9f298cd1659-kube-api-access-4nfht\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.827218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.827203 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.928171 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.928140 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nfht\" (UniqueName: \"kubernetes.io/projected/a29ea145-1600-4f8c-b46e-a9f298cd1659-kube-api-access-4nfht\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.928334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.928187 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.928334 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.928232 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.928954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.928936 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.930633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.930613 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:25.937418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:25.937394 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nfht\" (UniqueName: \"kubernetes.io/projected/a29ea145-1600-4f8c-b46e-a9f298cd1659-kube-api-access-4nfht\") pod \"maas-api-revoke-sub-d576e68b-wpw6s\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:26.017996 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:26.017894 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:26.291684 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:26.291652 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s"] Nov 01 16:45:26.293287 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:45:26.293258 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29ea145_1600_4f8c_b46e_a9f298cd1659.slice/crio-e181a547425e1c79ece04f22c37200c17bc7dd8661d383dd7356f30c4c3ea3db WatchSource:0}: Error finding container e181a547425e1c79ece04f22c37200c17bc7dd8661d383dd7356f30c4c3ea3db: Status 404 returned error can't find the container with id e181a547425e1c79ece04f22c37200c17bc7dd8661d383dd7356f30c4c3ea3db Nov 01 16:45:26.455613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:26.455582 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b617307-2229-4e9d-a9d5-72140bc298df" path="/var/lib/kubelet/pods/2b617307-2229-4e9d-a9d5-72140bc298df/volumes" Nov 01 16:45:26.771956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:26.771917 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" event={"ID":"a29ea145-1600-4f8c-b46e-a9f298cd1659","Type":"ContainerStarted","Data":"6179743cb95528662aade88cc0054936fdce6a9f3bd3b36bccb206037715ee51"} Nov 01 16:45:26.772399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:26.771973 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" event={"ID":"a29ea145-1600-4f8c-b46e-a9f298cd1659","Type":"ContainerStarted","Data":"e181a547425e1c79ece04f22c37200c17bc7dd8661d383dd7356f30c4c3ea3db"} Nov 01 16:45:26.787258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:26.787208 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" podStartSLOduration=1.787192078 podStartE2EDuration="1.787192078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:45:26.787186036 +0000 UTC m=+3464.490351971" watchObservedRunningTime="2026-11-01 16:45:26.787192078 +0000 UTC m=+3464.490358016" Nov 01 16:45:27.779131 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:27.779095 4573 generic.go:356] "Generic (PLEG): container finished" podID="a29ea145-1600-4f8c-b46e-a9f298cd1659" containerID="6179743cb95528662aade88cc0054936fdce6a9f3bd3b36bccb206037715ee51" exitCode=0 Nov 01 16:45:27.779565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:27.779175 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" event={"ID":"a29ea145-1600-4f8c-b46e-a9f298cd1659","Type":"ContainerDied","Data":"6179743cb95528662aade88cc0054936fdce6a9f3bd3b36bccb206037715ee51"} Nov 01 16:45:28.918227 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:28.918200 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:29.055966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.055877 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-openshift-ca\") pod \"a29ea145-1600-4f8c-b46e-a9f298cd1659\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " Nov 01 16:45:29.056146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.055982 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29ea145-1600-4f8c-b46e-a9f298cd1659-kube-api-access-4nfht\" (UniqueName: \"kubernetes.io/projected/a29ea145-1600-4f8c-b46e-a9f298cd1659-kube-api-access-4nfht\") pod \"a29ea145-1600-4f8c-b46e-a9f298cd1659\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " Nov 01 16:45:29.056146 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.056022 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-xks-ca\") pod \"a29ea145-1600-4f8c-b46e-a9f298cd1659\" (UID: \"a29ea145-1600-4f8c-b46e-a9f298cd1659\") " Nov 01 16:45:29.056293 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.056276 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-openshift-ca" pod "a29ea145-1600-4f8c-b46e-a9f298cd1659" (UID: "a29ea145-1600-4f8c-b46e-a9f298cd1659"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:45:29.058134 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.058108 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-xks-ca" pod "a29ea145-1600-4f8c-b46e-a9f298cd1659" (UID: "a29ea145-1600-4f8c-b46e-a9f298cd1659"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:45:29.058274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.058255 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29ea145-1600-4f8c-b46e-a9f298cd1659-kube-api-access-4nfht" pod "a29ea145-1600-4f8c-b46e-a9f298cd1659" (UID: "a29ea145-1600-4f8c-b46e-a9f298cd1659"). InnerVolumeSpecName "kube-api-access-4nfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:45:29.157833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.157791 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nfht\" (UniqueName: \"kubernetes.io/projected/a29ea145-1600-4f8c-b46e-a9f298cd1659-kube-api-access-4nfht\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:45:29.157833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.157828 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:45:29.157833 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.157844 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29ea145-1600-4f8c-b46e-a9f298cd1659-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:45:29.790468 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.790439 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" Nov 01 16:45:29.790651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.790446 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s" event={"ID":"a29ea145-1600-4f8c-b46e-a9f298cd1659","Type":"ContainerDied","Data":"e181a547425e1c79ece04f22c37200c17bc7dd8661d383dd7356f30c4c3ea3db"} Nov 01 16:45:29.790651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:29.790553 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e181a547425e1c79ece04f22c37200c17bc7dd8661d383dd7356f30c4c3ea3db" Nov 01 16:45:33.101145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:33.101109 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:45:33.202782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:33.202732 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:45:36.556269 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:36.556233 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:45:41.657965 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:41.657934 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:45:43.022120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:43.022081 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs"] Nov 01 16:45:43.024644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:43.024615 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7dkqs"] Nov 01 16:45:44.456737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.456692 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2b7f712-7b38-480d-8bbb-5de26ace1266" path="/var/lib/kubelet/pods/b2b7f712-7b38-480d-8bbb-5de26ace1266/volumes" Nov 01 16:45:44.567056 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.567024 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt"] Nov 01 16:45:44.567508 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.567490 4573 cpu_manager.go:414] "containerMap: removing container" podUID="a29ea145-1600-4f8c-b46e-a9f298cd1659" containerName="revoke-keys" Nov 01 16:45:44.567601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.567511 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29ea145-1600-4f8c-b46e-a9f298cd1659" containerName="revoke-keys" Nov 01 16:45:44.567665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.567645 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29ea145-1600-4f8c-b46e-a9f298cd1659" containerName="revoke-keys" Nov 01 16:45:44.567665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.567659 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff45c06b-5eb5-42d6-9793-dabf9f642465" containerName="cleanup" Nov 01 16:45:44.572145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.572121 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.576025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.575987 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:45:44.576174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.576062 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:45:44.577933 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.577905 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt"] Nov 01 16:45:44.632824 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.632796 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.633023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.632971 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.633100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.633029 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56kv\" (UniqueName: \"kubernetes.io/projected/886b1450-86f7-4cdf-b704-c083f03bd8ff-kube-api-access-k56kv\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.734487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.734400 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56kv\" (UniqueName: \"kubernetes.io/projected/886b1450-86f7-4cdf-b704-c083f03bd8ff-kube-api-access-k56kv\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.734487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.734452 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.734683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.734544 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.735290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.735263 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.737152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.737123 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.742453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.742434 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56kv\" (UniqueName: \"kubernetes.io/projected/886b1450-86f7-4cdf-b704-c083f03bd8ff-kube-api-access-k56kv\") pod \"maas-api-revoke-sub-d2bc93bc-hr7jt\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:44.889031 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:44.889004 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:45.164591 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:45.164543 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt"] Nov 01 16:45:45.168873 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:45:45.168838 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886b1450_86f7_4cdf_b704_c083f03bd8ff.slice/crio-2dcfe3956ab19daf37a5d8896f63f78d867ad4d56b25e56b29464b7dc8510e32 WatchSource:0}: Error finding container 2dcfe3956ab19daf37a5d8896f63f78d867ad4d56b25e56b29464b7dc8510e32: Status 404 returned error can't find the container with id 2dcfe3956ab19daf37a5d8896f63f78d867ad4d56b25e56b29464b7dc8510e32 Nov 01 16:45:45.573066 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:45.573038 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:45:45.879183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:45.879141 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" event={"ID":"886b1450-86f7-4cdf-b704-c083f03bd8ff","Type":"ContainerStarted","Data":"e189c0c6e695719ea668689d24ebbf4c5a13c14f7c20eb92ba33f94a97db6019"} Nov 01 16:45:45.879363 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:45.879205 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" event={"ID":"886b1450-86f7-4cdf-b704-c083f03bd8ff","Type":"ContainerStarted","Data":"2dcfe3956ab19daf37a5d8896f63f78d867ad4d56b25e56b29464b7dc8510e32"} Nov 01 16:45:45.894341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:45.894292 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" podStartSLOduration=1.894279708 podStartE2EDuration="1.894279708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:45:45.892896741 +0000 UTC m=+3483.596062674" watchObservedRunningTime="2026-11-01 16:45:45.894279708 +0000 UTC m=+3483.597445659" Nov 01 16:45:46.885628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:46.885594 4573 generic.go:356] "Generic (PLEG): container finished" podID="886b1450-86f7-4cdf-b704-c083f03bd8ff" containerID="e189c0c6e695719ea668689d24ebbf4c5a13c14f7c20eb92ba33f94a97db6019" exitCode=0 Nov 01 16:45:46.886151 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:46.885677 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" event={"ID":"886b1450-86f7-4cdf-b704-c083f03bd8ff","Type":"ContainerDied","Data":"e189c0c6e695719ea668689d24ebbf4c5a13c14f7c20eb92ba33f94a97db6019"} Nov 01 16:45:48.024271 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.024243 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:48.067861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.067834 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-openshift-ca\") pod \"886b1450-86f7-4cdf-b704-c083f03bd8ff\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " Nov 01 16:45:48.068046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.067880 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-xks-ca\") pod \"886b1450-86f7-4cdf-b704-c083f03bd8ff\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " Nov 01 16:45:48.068046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.067916 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886b1450-86f7-4cdf-b704-c083f03bd8ff-kube-api-access-k56kv\" (UniqueName: \"kubernetes.io/projected/886b1450-86f7-4cdf-b704-c083f03bd8ff-kube-api-access-k56kv\") pod \"886b1450-86f7-4cdf-b704-c083f03bd8ff\" (UID: \"886b1450-86f7-4cdf-b704-c083f03bd8ff\") " Nov 01 16:45:48.068251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.068222 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-openshift-ca" pod "886b1450-86f7-4cdf-b704-c083f03bd8ff" (UID: "886b1450-86f7-4cdf-b704-c083f03bd8ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:45:48.069910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.069884 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886b1450-86f7-4cdf-b704-c083f03bd8ff-kube-api-access-k56kv" pod "886b1450-86f7-4cdf-b704-c083f03bd8ff" (UID: "886b1450-86f7-4cdf-b704-c083f03bd8ff"). InnerVolumeSpecName "kube-api-access-k56kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:45:48.070027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.069917 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-xks-ca" pod "886b1450-86f7-4cdf-b704-c083f03bd8ff" (UID: "886b1450-86f7-4cdf-b704-c083f03bd8ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:45:48.169772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.169658 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:45:48.169772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.169691 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886b1450-86f7-4cdf-b704-c083f03bd8ff-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:45:48.169772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.169701 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56kv\" (UniqueName: \"kubernetes.io/projected/886b1450-86f7-4cdf-b704-c083f03bd8ff-kube-api-access-k56kv\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:45:48.896876 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.896848 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" Nov 01 16:45:48.897051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.896884 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt" event={"ID":"886b1450-86f7-4cdf-b704-c083f03bd8ff","Type":"ContainerDied","Data":"2dcfe3956ab19daf37a5d8896f63f78d867ad4d56b25e56b29464b7dc8510e32"} Nov 01 16:45:48.897051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:48.896931 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dcfe3956ab19daf37a5d8896f63f78d867ad4d56b25e56b29464b7dc8510e32" Nov 01 16:45:50.381378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:50.381345 4573 scope.go:122] "RemoveContainer" containerID="267fe5f87888238f906169e551b2dd67eb201a61c79310b3caddaf2de3d7e9bb" Nov 01 16:45:50.390106 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:50.390086 4573 scope.go:122] "RemoveContainer" containerID="54b7f7c8fde255e8fe9b14025a8611c0f10b8076cd344ed6cb496fb5851bc62a" Nov 01 16:45:50.397705 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:50.397689 4573 scope.go:122] "RemoveContainer" containerID="86de13504acef3f35fe9786d23fa7cf24106d972afabadb7d9e36f666c1a925b" Nov 01 16:45:54.362226 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:45:54.362190 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:45:55.476314 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:55.476279 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:45:56.141383 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:56.141346 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:45:56.670984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:56.670947 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:45:56.880987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:45:56.880950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:00.105818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:00.105782 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:00.205455 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:00.205421 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:01.252343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.252253 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7"] Nov 01 16:46:01.252925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.252867 4573 cpu_manager.go:414] "containerMap: removing container" podUID="886b1450-86f7-4cdf-b704-c083f03bd8ff" containerName="revoke-keys" Nov 01 16:46:01.252925 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.252887 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886b1450-86f7-4cdf-b704-c083f03bd8ff" containerName="revoke-keys" Nov 01 16:46:01.253119 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.252997 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="886b1450-86f7-4cdf-b704-c083f03bd8ff" containerName="revoke-keys" Nov 01 16:46:01.258169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.258144 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.274963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.274218 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-230d0a-serving-cert\"" Nov 01 16:46:01.275303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.275273 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 16:46:01.279710 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.279676 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7"] Nov 01 16:46:01.279877 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.279839 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 16:46:01.305278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.305241 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:01.389175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.389139 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:01.398494 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.398463 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.398649 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.398537 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59wjg\" (UniqueName: \"kubernetes.io/projected/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-kube-api-access-59wjg\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.398701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.398645 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.398701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.398683 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.457512 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.457475 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k"] Nov 01 16:46:01.461981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.461950 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:01.467579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.467547 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k"] Nov 01 16:46:01.485769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.481315 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:01.499283 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.499251 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.499439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.499303 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.499439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.499330 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.499439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.499381 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59wjg\" (UniqueName: \"kubernetes.io/projected/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-kube-api-access-59wjg\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.500969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.500341 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.503007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.502924 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.503672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.503623 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.509192 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.509139 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59wjg\" (UniqueName: \"kubernetes.io/projected/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-kube-api-access-59wjg\") pod \"maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.530207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.530166 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-230d0a\"" Nov 01 16:46:01.530797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.530774 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-785e6019-dockercfg-ckcnl\"" Nov 01 16:46:01.576198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.576163 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:01.600721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.600687 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv8p5\" (UniqueName: \"kubernetes.io/projected/2e4a0b5a-d452-4036-9a48-f92201619ffc-kube-api-access-xv8p5\") pod \"payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k\" (UID: \"2e4a0b5a-d452-4036-9a48-f92201619ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:01.600912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.600857 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a0b5a-d452-4036-9a48-f92201619ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k\" (UID: \"2e4a0b5a-d452-4036-9a48-f92201619ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:01.701923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.701890 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a0b5a-d452-4036-9a48-f92201619ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k\" (UID: \"2e4a0b5a-d452-4036-9a48-f92201619ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:01.702103 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.701977 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8p5\" (UniqueName: \"kubernetes.io/projected/2e4a0b5a-d452-4036-9a48-f92201619ffc-kube-api-access-xv8p5\") pod \"payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k\" (UID: \"2e4a0b5a-d452-4036-9a48-f92201619ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:01.702579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.702558 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a0b5a-d452-4036-9a48-f92201619ffc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k\" (UID: \"2e4a0b5a-d452-4036-9a48-f92201619ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:01.710498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.710466 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8p5\" (UniqueName: \"kubernetes.io/projected/2e4a0b5a-d452-4036-9a48-f92201619ffc-kube-api-access-xv8p5\") pod \"payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k\" (UID: \"2e4a0b5a-d452-4036-9a48-f92201619ffc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:01.782152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.782065 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:01.873983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.873942 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7"] Nov 01 16:46:01.875538 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:46:01.875510 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbddcd9a2_29d7_4319_b311_dc1aeda6caa4.slice/crio-e6292df90cf016ccb1026fe6fe9e7c1a6705410e5a3bce87cb7d35b3b0e18873 WatchSource:0}: Error finding container e6292df90cf016ccb1026fe6fe9e7c1a6705410e5a3bce87cb7d35b3b0e18873: Status 404 returned error can't find the container with id e6292df90cf016ccb1026fe6fe9e7c1a6705410e5a3bce87cb7d35b3b0e18873 Nov 01 16:46:01.965185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:01.965136 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" event={"ID":"bddcd9a2-29d7-4319-b311-dc1aeda6caa4","Type":"ContainerStarted","Data":"e6292df90cf016ccb1026fe6fe9e7c1a6705410e5a3bce87cb7d35b3b0e18873"} Nov 01 16:46:02.057957 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.057927 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k"] Nov 01 16:46:02.059659 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:46:02.059634 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4a0b5a_d452_4036_9a48_f92201619ffc.slice/crio-df2c077756c6b624479be5acdf9ae316118c01d7a21d84ddc942eb57a1b77030 WatchSource:0}: Error finding container df2c077756c6b624479be5acdf9ae316118c01d7a21d84ddc942eb57a1b77030: Status 404 returned error can't find the container with id df2c077756c6b624479be5acdf9ae316118c01d7a21d84ddc942eb57a1b77030 Nov 01 16:46:02.191887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.191851 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:02.793288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.793254 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:02.972447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.972406 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" event={"ID":"bddcd9a2-29d7-4319-b311-dc1aeda6caa4","Type":"ContainerStarted","Data":"e266e5fa67818fc3d998dc8463e4201e7aa8f84a5c08a6772d92329f02a5de3f"} Nov 01 16:46:02.972647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.972468 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:02.974091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.974064 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" event={"ID":"2e4a0b5a-d452-4036-9a48-f92201619ffc","Type":"ContainerStarted","Data":"c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce"} Nov 01 16:46:02.974201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.974111 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" event={"ID":"2e4a0b5a-d452-4036-9a48-f92201619ffc","Type":"ContainerStarted","Data":"df2c077756c6b624479be5acdf9ae316118c01d7a21d84ddc942eb57a1b77030"} Nov 01 16:46:02.974201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.974179 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:02.989258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:02.989216 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" podStartSLOduration=1.5134342699999999 podStartE2EDuration="1.989204349s" totalImagesPullingTime="475.770079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:46:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:46:02.988624857 +0000 UTC m=+3500.691790793" watchObservedRunningTime="2026-11-01 16:46:02.989204349 +0000 UTC m=+3500.692370284" Nov 01 16:46:03.004630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:03.004587 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" podStartSLOduration=2.004572645 podStartE2EDuration="2.004572645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:46:03.002690097 +0000 UTC m=+3500.705856040" watchObservedRunningTime="2026-11-01 16:46:03.004572645 +0000 UTC m=+3500.707738582" Nov 01 16:46:03.406568 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:03.406530 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:03.770909 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:03.770817 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:46:04.101460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:04.101384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:04.700045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:04.700011 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:05.385892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:05.385860 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:06.012606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:06.012543 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:06.678395 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:06.678353 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:07.297466 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:07.297429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:07.996260 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:07.996226 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:08.585690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:08.585658 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:08.985572 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:08.985541 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:46:09.193444 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:09.193403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:09.854496 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:09.854459 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:10.572901 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:10.572866 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:11.192409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:11.192358 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:11.796683 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:11.796647 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:12.468894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:12.468858 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:13.098633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:13.098598 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:13.750259 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:13.750219 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:13.983164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:13.983133 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:46:14.372682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:14.372650 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:15.042570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:15.042538 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:15.142327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:15.142296 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:15.743276 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:15.743242 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:15.917986 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:15.917924 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:16.054452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:16.054360 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:16.588070 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:16.588038 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:17.068281 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:17.068248 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:17.394290 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:17.394260 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:17.988652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:17.988619 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:18.664910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:18.664878 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:46:18.700161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:18.700133 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:19.471199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:19.471167 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:20.082372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:20.082339 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:20.785609 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:20.785577 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:20.995074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:20.995038 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:21.095149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:21.095072 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:21.488142 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:21.488107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:22.099971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:22.099925 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:22.792408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:22.792374 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:23.484652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:23.484618 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:24.086262 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:24.086229 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:24.363181 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:46:24.363150 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:46:24.691529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:24.691442 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:25.451847 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:25.451790 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:26.069287 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:26.069255 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:26.782379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:26.782349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:27.400045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:27.400005 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:28.060659 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:28.060625 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:28.676034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:28.675995 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:29.287125 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:29.287095 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:29.886570 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:29.886539 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:30.490044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:30.490001 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:31.088172 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:31.088136 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:31.685664 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:31.685629 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:32.289982 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:32.289940 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:33.056969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:33.056940 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:33.158513 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:33.158483 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:33.709051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:33.709009 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:34.404870 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:34.404840 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:35.041669 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:35.041638 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:35.644701 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:35.644669 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:46:35.684885 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:35.684848 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:36.481536 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:36.481503 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:37.093541 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:37.093510 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:37.688908 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:37.688875 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:38.287586 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:38.287555 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:38.892061 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:38.892031 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:39.585415 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:39.585384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:40.020410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:40.020378 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb"] Nov 01 16:46:40.024441 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:40.024412 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-4zppb"] Nov 01 16:46:40.192100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:40.192063 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:40.456000 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:40.455961 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dc64f78-4c11-471a-942b-97df6de578ec" path="/var/lib/kubelet/pods/6dc64f78-4c11-471a-942b-97df6de578ec/volumes" Nov 01 16:46:40.802785 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:40.802689 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:41.460470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:41.460441 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:42.077486 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:42.077452 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:42.682358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:42.682323 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:43.281340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:43.281294 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:43.893700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:43.893667 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:44.494866 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:44.494834 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:45.085827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:45.085792 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:45.888093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:45.888060 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:46.496601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:46.496564 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:47.097608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:47.097577 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:47.798968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:47.798936 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:48.417585 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:48.417553 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:49.103556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:49.103523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:49.763539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:49.763481 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:50.022136 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:50.021977 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76"] Nov 01 16:46:50.030034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:50.025826 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-c6b76"] Nov 01 16:46:50.431407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:50.431366 4573 scope.go:122] "RemoveContainer" containerID="87e6ddfb582e9698d215659a0fbfa2a12cedcb9eaed84bb9ce7d066629900c2b" Nov 01 16:46:50.442173 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:50.442152 4573 scope.go:122] "RemoveContainer" containerID="9868e7756b9e3ba66a046158d5c506a6f5c4ba3a13defc1ed11fa939480991e9" Nov 01 16:46:50.452329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:50.452309 4573 scope.go:122] "RemoveContainer" containerID="f4361abd14e230124a8ee330938ddd6b94bb4cd102d3290d5fb0046fcd8641b4" Nov 01 16:46:50.457410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:50.457386 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3199fff-ff05-46cb-bd9a-981fcfd06599" path="/var/lib/kubelet/pods/e3199fff-ff05-46cb-bd9a-981fcfd06599/volumes" Nov 01 16:46:50.464325 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:50.464304 4573 scope.go:122] "RemoveContainer" containerID="4370146b6f4a38965c2e365d69193b167ed055ea5d97061d495534711eea570c" Nov 01 16:46:50.482761 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:50.482720 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:51.086995 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:51.086964 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:51.684942 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:51.684904 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:52.286534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:52.286506 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:52.981861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:52.981827 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:53.578873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:53.578835 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:54.188453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:54.188416 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:54.363118 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:46:54.363084 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:46:54.789128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:54.789090 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:55.402403 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:55.402356 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:56.097162 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:56.097107 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:56.693810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:56.693778 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:57.391843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:57.391810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:57.921236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:57.921202 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:58.021728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:58.021697 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:58.339576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:58.339491 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:46:58.896058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:58.896030 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:46:59.496992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:46:59.496956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:00.103152 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:00.103072 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:00.769624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:00.769590 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:01.394635 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:01.394599 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:01.984776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:01.984726 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:02.612094 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:02.612058 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:02.982002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:02.981916 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:03.184517 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:03.184457 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:03.872728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:03.872654 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:04.382805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:04.382769 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:05.087092 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:05.087056 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:05.780368 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:05.780335 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:06.480960 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:06.480926 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:07.082027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:07.081991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:07.782938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:07.782894 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:08.183005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:08.182967 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k"] Nov 01 16:47:08.237888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:08.237850 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:08.335124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:08.335088 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:08.508418 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:08.508332 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:08.974573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:08.974541 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7"] Nov 01 16:47:08.974796 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:08.974763 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" podUID="bddcd9a2-29d7-4319-b311-dc1aeda6caa4" containerName="maas-api" containerID="cri-o://e266e5fa67818fc3d998dc8463e4201e7aa8f84a5c08a6772d92329f02a5de3f" gracePeriod=30 Nov 01 16:47:08.979953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:08.979920 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" podUID="bddcd9a2-29d7-4319-b311-dc1aeda6caa4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.95:8443/health\": dial tcp 10.133.1.95:8443: connect: connection refused" Nov 01 16:47:09.018598 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.018563 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:09.141528 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.141490 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:09.217558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.217522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:09.331219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.331190 4573 generic.go:356] "Generic (PLEG): container finished" podID="bddcd9a2-29d7-4319-b311-dc1aeda6caa4" containerID="e266e5fa67818fc3d998dc8463e4201e7aa8f84a5c08a6772d92329f02a5de3f" exitCode=0 Nov 01 16:47:09.331365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.331270 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" event={"ID":"bddcd9a2-29d7-4319-b311-dc1aeda6caa4","Type":"ContainerDied","Data":"e266e5fa67818fc3d998dc8463e4201e7aa8f84a5c08a6772d92329f02a5de3f"} Nov 01 16:47:09.424036 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.424012 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:47:09.473565 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.473533 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-maas-api-tls\") pod \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " Nov 01 16:47:09.473732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.473664 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-metrics-certs\") pod \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " Nov 01 16:47:09.473842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.473729 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-openshift-service-ca\") pod \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " Nov 01 16:47:09.473842 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.473815 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-kube-api-access-59wjg\" (UniqueName: \"kubernetes.io/projected/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-kube-api-access-59wjg\") pod \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\" (UID: \"bddcd9a2-29d7-4319-b311-dc1aeda6caa4\") " Nov 01 16:47:09.474120 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.474081 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-openshift-service-ca" pod "bddcd9a2-29d7-4319-b311-dc1aeda6caa4" (UID: "bddcd9a2-29d7-4319-b311-dc1aeda6caa4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:47:09.474254 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.474213 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:09.475760 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.475727 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-maas-api-tls" pod "bddcd9a2-29d7-4319-b311-dc1aeda6caa4" (UID: "bddcd9a2-29d7-4319-b311-dc1aeda6caa4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:47:09.476288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.476263 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-metrics-certs" pod "bddcd9a2-29d7-4319-b311-dc1aeda6caa4" (UID: "bddcd9a2-29d7-4319-b311-dc1aeda6caa4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:47:09.476388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.476308 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-kube-api-access-59wjg" pod "bddcd9a2-29d7-4319-b311-dc1aeda6caa4" (UID: "bddcd9a2-29d7-4319-b311-dc1aeda6caa4"). InnerVolumeSpecName "kube-api-access-59wjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:47:09.575034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.574952 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:09.575034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.574978 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59wjg\" (UniqueName: \"kubernetes.io/projected/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-kube-api-access-59wjg\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:09.575034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.574989 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bddcd9a2-29d7-4319-b311-dc1aeda6caa4-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:09.743386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:09.743356 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:10.337744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:10.337664 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" Nov 01 16:47:10.337744 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:10.337689 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7" event={"ID":"bddcd9a2-29d7-4319-b311-dc1aeda6caa4","Type":"ContainerDied","Data":"e6292df90cf016ccb1026fe6fe9e7c1a6705410e5a3bce87cb7d35b3b0e18873"} Nov 01 16:47:10.338249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:10.337775 4573 scope.go:122] "RemoveContainer" containerID="e266e5fa67818fc3d998dc8463e4201e7aa8f84a5c08a6772d92329f02a5de3f" Nov 01 16:47:10.359087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:10.359058 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7"] Nov 01 16:47:10.366163 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:10.366141 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-230d0a-58c5456886-jscc7"] Nov 01 16:47:10.457700 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:10.457666 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bddcd9a2-29d7-4319-b311-dc1aeda6caa4" path="/var/lib/kubelet/pods/bddcd9a2-29d7-4319-b311-dc1aeda6caa4/volumes" Nov 01 16:47:10.735374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:10.735338 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:47:13.184277 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:13.184217 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" podUID="2e4a0b5a-d452-4036-9a48-f92201619ffc" containerName="payload-pre-processing" containerID="cri-o://c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce" gracePeriod=25 Nov 01 16:47:13.980898 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:13.980857 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" podUID="2e4a0b5a-d452-4036-9a48-f92201619ffc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:47:15.497467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:15.497430 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:15.571203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:15.571167 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:15.852296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:15.852215 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:20.514963 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:20.514930 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:22.050174 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:22.050135 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:47:23.999775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:23.998860 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" podUID="2e4a0b5a-d452-4036-9a48-f92201619ffc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 16:47:24.393461 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:47:24.393379 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:47:33.153622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.153584 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:33.435378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.435348 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:47:33.463973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.463939 4573 generic.go:356] "Generic (PLEG): container finished" podID="2e4a0b5a-d452-4036-9a48-f92201619ffc" containerID="c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce" exitCode=0 Nov 01 16:47:33.464147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.464001 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" Nov 01 16:47:33.464147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.464017 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" event={"ID":"2e4a0b5a-d452-4036-9a48-f92201619ffc","Type":"ContainerDied","Data":"c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce"} Nov 01 16:47:33.464147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.464078 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k" event={"ID":"2e4a0b5a-d452-4036-9a48-f92201619ffc","Type":"ContainerDied","Data":"df2c077756c6b624479be5acdf9ae316118c01d7a21d84ddc942eb57a1b77030"} Nov 01 16:47:33.464147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.464110 4573 scope.go:122] "RemoveContainer" containerID="c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce" Nov 01 16:47:33.473713 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.473691 4573 scope.go:122] "RemoveContainer" containerID="c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce" Nov 01 16:47:33.474003 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:47:33.473984 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce\": container with ID starting with c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce not found: ID does not exist" containerID="c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce" Nov 01 16:47:33.474075 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.474012 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce"} err="failed to get container status \"c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce\": rpc error: code = NotFound desc = could not find container \"c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce\": container with ID starting with c323540175d184bea2946c3e19c6d96fe493afc6564cb206867d016cb8e2b7ce not found: ID does not exist" Nov 01 16:47:33.514399 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.514375 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4a0b5a-d452-4036-9a48-f92201619ffc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a0b5a-d452-4036-9a48-f92201619ffc-plugins-config-volume\") pod \"2e4a0b5a-d452-4036-9a48-f92201619ffc\" (UID: \"2e4a0b5a-d452-4036-9a48-f92201619ffc\") " Nov 01 16:47:33.514542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.514439 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4a0b5a-d452-4036-9a48-f92201619ffc-kube-api-access-xv8p5\" (UniqueName: \"kubernetes.io/projected/2e4a0b5a-d452-4036-9a48-f92201619ffc-kube-api-access-xv8p5\") pod \"2e4a0b5a-d452-4036-9a48-f92201619ffc\" (UID: \"2e4a0b5a-d452-4036-9a48-f92201619ffc\") " Nov 01 16:47:33.514778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.514747 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4a0b5a-d452-4036-9a48-f92201619ffc-plugins-config-volume" pod "2e4a0b5a-d452-4036-9a48-f92201619ffc" (UID: "2e4a0b5a-d452-4036-9a48-f92201619ffc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:47:33.516461 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.516439 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4a0b5a-d452-4036-9a48-f92201619ffc-kube-api-access-xv8p5" pod "2e4a0b5a-d452-4036-9a48-f92201619ffc" (UID: "2e4a0b5a-d452-4036-9a48-f92201619ffc"). InnerVolumeSpecName "kube-api-access-xv8p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:47:33.615889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.615847 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4a0b5a-d452-4036-9a48-f92201619ffc-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:33.615889 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.615884 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv8p5\" (UniqueName: \"kubernetes.io/projected/2e4a0b5a-d452-4036-9a48-f92201619ffc-kube-api-access-xv8p5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:33.787447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.787414 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k"] Nov 01 16:47:33.793718 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:33.793692 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-230d0a-f4cc6zl88k"] Nov 01 16:47:34.456989 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:34.456956 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4a0b5a-d452-4036-9a48-f92201619ffc" path="/var/lib/kubelet/pods/2e4a0b5a-d452-4036-9a48-f92201619ffc/volumes" Nov 01 16:47:36.540849 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:36.540812 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:38.174629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:38.174593 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:47:39.790485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:39.790446 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:40.450661 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:40.450626 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:43.477711 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:43.477684 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:47:43.489372 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:43.489345 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:47:43.491046 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:43.491024 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:47:43.503462 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:43.503443 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:47:46.077954 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:46.077919 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:47:50.501140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:50.501113 4573 scope.go:122] "RemoveContainer" containerID="d3e72cab8462cf7d2a51d89b3cdc6bc2c321b54214be3d69c3f0aef3d4298816" Nov 01 16:47:50.510045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:50.510023 4573 scope.go:122] "RemoveContainer" containerID="c69ae1bdaa76bad7a9fe18a992718c87df716a92f485f8ceccaa4e8814be62d1" Nov 01 16:47:50.518017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:50.517999 4573 scope.go:122] "RemoveContainer" containerID="85cd21520bebc63a32a546f98a08146c5d942adbb62f398b437f0564ba42ec97" Nov 01 16:47:53.442194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.442154 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r"] Nov 01 16:47:53.442662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.442571 4573 cpu_manager.go:414] "containerMap: removing container" podUID="2e4a0b5a-d452-4036-9a48-f92201619ffc" containerName="payload-pre-processing" Nov 01 16:47:53.442662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.442582 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4a0b5a-d452-4036-9a48-f92201619ffc" containerName="payload-pre-processing" Nov 01 16:47:53.442662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.442606 4573 cpu_manager.go:414] "containerMap: removing container" podUID="bddcd9a2-29d7-4319-b311-dc1aeda6caa4" containerName="maas-api" Nov 01 16:47:53.442662 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.442611 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bddcd9a2-29d7-4319-b311-dc1aeda6caa4" containerName="maas-api" Nov 01 16:47:53.442855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.442674 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4a0b5a-d452-4036-9a48-f92201619ffc" containerName="payload-pre-processing" Nov 01 16:47:53.442855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.442685 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="bddcd9a2-29d7-4319-b311-dc1aeda6caa4" containerName="maas-api" Nov 01 16:47:53.447514 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.447497 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.450970 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.450944 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:47:53.451168 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.451151 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:47:53.452900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.452874 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r"] Nov 01 16:47:53.537105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.537072 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.537307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.537215 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvfx\" (UniqueName: \"kubernetes.io/projected/5a0bc57b-b9f4-4877-b458-a1cd20830f23-kube-api-access-4gvfx\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.537307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.537295 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.638539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.638504 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.638734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.638567 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvfx\" (UniqueName: \"kubernetes.io/projected/5a0bc57b-b9f4-4877-b458-a1cd20830f23-kube-api-access-4gvfx\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.638734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.638627 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.639327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.639300 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.640879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.640852 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.645910 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.645888 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvfx\" (UniqueName: \"kubernetes.io/projected/5a0bc57b-b9f4-4877-b458-a1cd20830f23-kube-api-access-4gvfx\") pod \"maas-api-revoke-sub-452b6d7b-dhl7r\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.765296 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.765222 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:53.959199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:53.959163 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:47:54.037096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:54.037017 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r"] Nov 01 16:47:54.039691 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:47:54.039664 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0bc57b_b9f4_4877_b458_a1cd20830f23.slice/crio-8e831b16b2f407dce167ad4af04545dd6f91861df686cc7cf3829834c0ab3f37 WatchSource:0}: Error finding container 8e831b16b2f407dce167ad4af04545dd6f91861df686cc7cf3829834c0ab3f37: Status 404 returned error can't find the container with id 8e831b16b2f407dce167ad4af04545dd6f91861df686cc7cf3829834c0ab3f37 Nov 01 16:47:54.376883 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:47:54.376834 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:47:54.601272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:54.601229 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" event={"ID":"5a0bc57b-b9f4-4877-b458-a1cd20830f23","Type":"ContainerStarted","Data":"8257d5d58034bab61d77545910c4b4ef0236f5a8ca668543f2c75306fcbe0651"} Nov 01 16:47:54.601676 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:54.601300 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" event={"ID":"5a0bc57b-b9f4-4877-b458-a1cd20830f23","Type":"ContainerStarted","Data":"8e831b16b2f407dce167ad4af04545dd6f91861df686cc7cf3829834c0ab3f37"} Nov 01 16:47:54.619347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:54.619292 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" podStartSLOduration=1.619273925 podStartE2EDuration="1.619273925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:47:54.616008124 +0000 UTC m=+3612.319174072" watchObservedRunningTime="2026-11-01 16:47:54.619273925 +0000 UTC m=+3612.322439863" Nov 01 16:47:55.620115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:55.620075 4573 generic.go:356] "Generic (PLEG): container finished" podID="5a0bc57b-b9f4-4877-b458-a1cd20830f23" containerID="8257d5d58034bab61d77545910c4b4ef0236f5a8ca668543f2c75306fcbe0651" exitCode=0 Nov 01 16:47:55.626777 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:55.620718 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" event={"ID":"5a0bc57b-b9f4-4877-b458-a1cd20830f23","Type":"ContainerDied","Data":"8257d5d58034bab61d77545910c4b4ef0236f5a8ca668543f2c75306fcbe0651"} Nov 01 16:47:57.040169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.040135 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:57.145836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.145793 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-xks-ca\") pod \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " Nov 01 16:47:57.146059 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.145888 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0bc57b-b9f4-4877-b458-a1cd20830f23-kube-api-access-4gvfx\" (UniqueName: \"kubernetes.io/projected/5a0bc57b-b9f4-4877-b458-a1cd20830f23-kube-api-access-4gvfx\") pod \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " Nov 01 16:47:57.146511 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.146485 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-openshift-ca\") pod \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\" (UID: \"5a0bc57b-b9f4-4877-b458-a1cd20830f23\") " Nov 01 16:47:57.146946 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.146898 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-openshift-ca" pod "5a0bc57b-b9f4-4877-b458-a1cd20830f23" (UID: "5a0bc57b-b9f4-4877-b458-a1cd20830f23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:47:57.147252 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.147226 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:57.148229 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.148204 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0bc57b-b9f4-4877-b458-a1cd20830f23-kube-api-access-4gvfx" pod "5a0bc57b-b9f4-4877-b458-a1cd20830f23" (UID: "5a0bc57b-b9f4-4877-b458-a1cd20830f23"). InnerVolumeSpecName "kube-api-access-4gvfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:47:57.148619 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.148593 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-xks-ca" pod "5a0bc57b-b9f4-4877-b458-a1cd20830f23" (UID: "5a0bc57b-b9f4-4877-b458-a1cd20830f23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:47:57.248795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.248677 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0bc57b-b9f4-4877-b458-a1cd20830f23-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:57.248795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.248711 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvfx\" (UniqueName: \"kubernetes.io/projected/5a0bc57b-b9f4-4877-b458-a1cd20830f23-kube-api-access-4gvfx\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:47:57.640477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.640447 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" Nov 01 16:47:57.640477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.640468 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r" event={"ID":"5a0bc57b-b9f4-4877-b458-a1cd20830f23","Type":"ContainerDied","Data":"8e831b16b2f407dce167ad4af04545dd6f91861df686cc7cf3829834c0ab3f37"} Nov 01 16:47:57.640695 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:47:57.640512 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e831b16b2f407dce167ad4af04545dd6f91861df686cc7cf3829834c0ab3f37" Nov 01 16:48:04.311484 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:04.311450 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:05.596644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:05.596610 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:05.677952 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:05.677918 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:48:06.489542 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.489506 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh"] Nov 01 16:48:06.490194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.490171 4573 cpu_manager.go:414] "containerMap: removing container" podUID="5a0bc57b-b9f4-4877-b458-a1cd20830f23" containerName="revoke-keys" Nov 01 16:48:06.490250 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.490200 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0bc57b-b9f4-4877-b458-a1cd20830f23" containerName="revoke-keys" Nov 01 16:48:06.490365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.490351 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0bc57b-b9f4-4877-b458-a1cd20830f23" containerName="revoke-keys" Nov 01 16:48:06.494507 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.494481 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.499879 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.497599 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh"] Nov 01 16:48:06.504045 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.504017 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:48:06.504194 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.504096 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:48:06.520193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.520163 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.520923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.520890 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.521052 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.520942 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4fp\" (UniqueName: \"kubernetes.io/projected/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-kube-api-access-fq4fp\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.622015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.621960 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.622407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.622064 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.622407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.622108 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4fp\" (UniqueName: \"kubernetes.io/projected/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-kube-api-access-fq4fp\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.622896 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.622860 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.624668 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.624639 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.630219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.630176 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4fp\" (UniqueName: \"kubernetes.io/projected/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-kube-api-access-fq4fp\") pod \"maas-api-revoke-sub-3e31a506-lnpbh\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:06.818100 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:06.818017 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:07.138707 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:07.138601 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh"] Nov 01 16:48:07.147379 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:48:07.147350 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b84bee_caa8_40d6_aaa3_ce08bd6798c5.slice/crio-41005896b89564ccd0c8b0e6569b3fc3b810db3ed7f9796532d26423f7a84fb0 WatchSource:0}: Error finding container 41005896b89564ccd0c8b0e6569b3fc3b810db3ed7f9796532d26423f7a84fb0: Status 404 returned error can't find the container with id 41005896b89564ccd0c8b0e6569b3fc3b810db3ed7f9796532d26423f7a84fb0 Nov 01 16:48:07.734234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:07.734012 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" event={"ID":"28b84bee-caa8-40d6-aaa3-ce08bd6798c5","Type":"ContainerStarted","Data":"d87ce00b2f5eaba43173fdb2098ace604a9a50c6b2e7502dd98d96c0659991ac"} Nov 01 16:48:07.734234 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:07.734116 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" event={"ID":"28b84bee-caa8-40d6-aaa3-ce08bd6798c5","Type":"ContainerStarted","Data":"41005896b89564ccd0c8b0e6569b3fc3b810db3ed7f9796532d26423f7a84fb0"} Nov 01 16:48:07.749354 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:07.749308 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" podStartSLOduration=1.7492948579999998 podStartE2EDuration="1.749294858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:07.747327161 +0000 UTC m=+3625.450493127" watchObservedRunningTime="2026-11-01 16:48:07.749294858 +0000 UTC m=+3625.452460824" Nov 01 16:48:08.743021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:08.742980 4573 generic.go:356] "Generic (PLEG): container finished" podID="28b84bee-caa8-40d6-aaa3-ce08bd6798c5" containerID="d87ce00b2f5eaba43173fdb2098ace604a9a50c6b2e7502dd98d96c0659991ac" exitCode=0 Nov 01 16:48:08.743452 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:08.743053 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" event={"ID":"28b84bee-caa8-40d6-aaa3-ce08bd6798c5","Type":"ContainerDied","Data":"d87ce00b2f5eaba43173fdb2098ace604a9a50c6b2e7502dd98d96c0659991ac"} Nov 01 16:48:09.869671 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.869644 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:09.884846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.884827 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:09.942257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.942223 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:09.967515 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.967491 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-xks-ca\") pod \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " Nov 01 16:48:09.967599 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.967585 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-openshift-ca\") pod \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " Nov 01 16:48:09.967640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.967610 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-kube-api-access-fq4fp\" (UniqueName: \"kubernetes.io/projected/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-kube-api-access-fq4fp\") pod \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\" (UID: \"28b84bee-caa8-40d6-aaa3-ce08bd6798c5\") " Nov 01 16:48:09.967950 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.967932 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-openshift-ca" pod "28b84bee-caa8-40d6-aaa3-ce08bd6798c5" (UID: "28b84bee-caa8-40d6-aaa3-ce08bd6798c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:48:09.969604 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.969578 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-xks-ca" pod "28b84bee-caa8-40d6-aaa3-ce08bd6798c5" (UID: "28b84bee-caa8-40d6-aaa3-ce08bd6798c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:48:09.969708 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:09.969642 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-kube-api-access-fq4fp" pod "28b84bee-caa8-40d6-aaa3-ce08bd6798c5" (UID: "28b84bee-caa8-40d6-aaa3-ce08bd6798c5"). InnerVolumeSpecName "kube-api-access-fq4fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:48:10.068894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:10.068795 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:10.068894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:10.068833 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:10.068894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:10.068850 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4fp\" (UniqueName: \"kubernetes.io/projected/28b84bee-caa8-40d6-aaa3-ce08bd6798c5-kube-api-access-fq4fp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:10.758844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:10.758680 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" event={"ID":"28b84bee-caa8-40d6-aaa3-ce08bd6798c5","Type":"ContainerDied","Data":"41005896b89564ccd0c8b0e6569b3fc3b810db3ed7f9796532d26423f7a84fb0"} Nov 01 16:48:10.758844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:10.758723 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh" Nov 01 16:48:10.758844 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:10.758744 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41005896b89564ccd0c8b0e6569b3fc3b810db3ed7f9796532d26423f7a84fb0" Nov 01 16:48:12.145366 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:12.145333 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:48:17.340562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:17.340516 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:48:23.093303 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:23.093268 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:48:24.365496 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:48:24.365416 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:48:26.777831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.777800 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7"] Nov 01 16:48:26.778247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.778223 4573 cpu_manager.go:414] "containerMap: removing container" podUID="28b84bee-caa8-40d6-aaa3-ce08bd6798c5" containerName="revoke-keys" Nov 01 16:48:26.778247 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.778234 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28b84bee-caa8-40d6-aaa3-ce08bd6798c5" containerName="revoke-keys" Nov 01 16:48:26.778327 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.778300 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="28b84bee-caa8-40d6-aaa3-ce08bd6798c5" containerName="revoke-keys" Nov 01 16:48:26.781682 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.781656 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:26.787581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.787551 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 16:48:26.788855 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.788572 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wthns\"" Nov 01 16:48:26.791207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.791181 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7"] Nov 01 16:48:26.981769 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.981722 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:26.981972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.981835 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbtp\" (UniqueName: \"kubernetes.io/projected/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-kube-api-access-8xbtp\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:26.981972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:26.981868 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:27.083288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.083201 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:27.083288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.083260 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbtp\" (UniqueName: \"kubernetes.io/projected/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-kube-api-access-8xbtp\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:27.083288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.083283 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:27.083955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.083932 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:27.085680 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.085656 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:27.090987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.090961 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbtp\" (UniqueName: \"kubernetes.io/projected/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-kube-api-access-8xbtp\") pod \"maas-api-revoke-sub-be80fab5-ncnd7\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:27.095625 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.095605 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:27.370360 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.370331 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7"] Nov 01 16:48:27.372516 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:48:27.372485 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84a0dc41_c45a_4e39_a7ac_53eb0186e00d.slice/crio-822e1cb442b1b18c2cd787932269b9bfb34b269b4b47444bc21edab2d4b174c6 WatchSource:0}: Error finding container 822e1cb442b1b18c2cd787932269b9bfb34b269b4b47444bc21edab2d4b174c6: Status 404 returned error can't find the container with id 822e1cb442b1b18c2cd787932269b9bfb34b269b4b47444bc21edab2d4b174c6 Nov 01 16:48:27.882343 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.882303 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" event={"ID":"84a0dc41-c45a-4e39-a7ac-53eb0186e00d","Type":"ContainerStarted","Data":"8db403080b0b3ef0b5120a373f73a7fb589cc8b59fe444127ef41b090844d761"} Nov 01 16:48:27.882874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.882367 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" event={"ID":"84a0dc41-c45a-4e39-a7ac-53eb0186e00d","Type":"ContainerStarted","Data":"822e1cb442b1b18c2cd787932269b9bfb34b269b4b47444bc21edab2d4b174c6"} Nov 01 16:48:27.902331 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:27.902281 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" podStartSLOduration=1.90226772 podStartE2EDuration="1.90226772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:27.900233999 +0000 UTC m=+3645.603399936" watchObservedRunningTime="2026-11-01 16:48:27.90226772 +0000 UTC m=+3645.605433653" Nov 01 16:48:28.888690 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:28.888652 4573 generic.go:356] "Generic (PLEG): container finished" podID="84a0dc41-c45a-4e39-a7ac-53eb0186e00d" containerID="8db403080b0b3ef0b5120a373f73a7fb589cc8b59fe444127ef41b090844d761" exitCode=0 Nov 01 16:48:28.889086 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:28.888734 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" event={"ID":"84a0dc41-c45a-4e39-a7ac-53eb0186e00d","Type":"ContainerDied","Data":"8db403080b0b3ef0b5120a373f73a7fb589cc8b59fe444127ef41b090844d761"} Nov 01 16:48:30.036025 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.035992 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:30.108539 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.108503 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-xks-ca\") pod \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " Nov 01 16:48:30.108719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.108555 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-openshift-ca\") pod \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " Nov 01 16:48:30.108719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.108605 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-kube-api-access-8xbtp\" (UniqueName: \"kubernetes.io/projected/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-kube-api-access-8xbtp\") pod \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\" (UID: \"84a0dc41-c45a-4e39-a7ac-53eb0186e00d\") " Nov 01 16:48:30.108958 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.108935 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-openshift-ca" pod "84a0dc41-c45a-4e39-a7ac-53eb0186e00d" (UID: "84a0dc41-c45a-4e39-a7ac-53eb0186e00d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:48:30.110471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.110449 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-xks-ca" pod "84a0dc41-c45a-4e39-a7ac-53eb0186e00d" (UID: "84a0dc41-c45a-4e39-a7ac-53eb0186e00d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:48:30.110610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.110589 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-kube-api-access-8xbtp" pod "84a0dc41-c45a-4e39-a7ac-53eb0186e00d" (UID: "84a0dc41-c45a-4e39-a7ac-53eb0186e00d"). InnerVolumeSpecName "kube-api-access-8xbtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:48:30.210274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.210184 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-xks-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:30.210274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.210215 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-maas-api-openshift-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:30.210274 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.210226 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbtp\" (UniqueName: \"kubernetes.io/projected/84a0dc41-c45a-4e39-a7ac-53eb0186e00d-kube-api-access-8xbtp\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:30.899388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.899362 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" Nov 01 16:48:30.899561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.899390 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7" event={"ID":"84a0dc41-c45a-4e39-a7ac-53eb0186e00d","Type":"ContainerDied","Data":"822e1cb442b1b18c2cd787932269b9bfb34b269b4b47444bc21edab2d4b174c6"} Nov 01 16:48:30.899561 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:30.899439 4573 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="822e1cb442b1b18c2cd787932269b9bfb34b269b4b47444bc21edab2d4b174c6" Nov 01 16:48:33.199156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:33.199121 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:37.668601 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:37.668558 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:48:38.937945 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.937913 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5"] Nov 01 16:48:38.938460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.938370 4573 cpu_manager.go:414] "containerMap: removing container" podUID="84a0dc41-c45a-4e39-a7ac-53eb0186e00d" containerName="revoke-keys" Nov 01 16:48:38.938460 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.938383 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a0dc41-c45a-4e39-a7ac-53eb0186e00d" containerName="revoke-keys" Nov 01 16:48:38.938576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.938476 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a0dc41-c45a-4e39-a7ac-53eb0186e00d" containerName="revoke-keys" Nov 01 16:48:38.941918 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.941896 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:38.950892 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.950865 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 16:48:38.951038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.950919 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 16:48:38.951038 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.951008 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l5k8h\"" Nov 01 16:48:38.957581 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:38.957555 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5"] Nov 01 16:48:39.000233 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.000190 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54gkm\" (UniqueName: \"kubernetes.io/projected/04f8f9f1-158f-4462-8dd5-061f94507728-kube-api-access-54gkm\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.000446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.000355 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-metrics-certs\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.000446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.000399 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f8f9f1-158f-4462-8dd5-061f94507728-openshift-service-ca\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.000578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.000494 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-maas-api-tls\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.101688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.101650 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-metrics-certs\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.101688 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.101692 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f8f9f1-158f-4462-8dd5-061f94507728-openshift-service-ca\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.101997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.101721 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-maas-api-tls\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.101997 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.101808 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54gkm\" (UniqueName: \"kubernetes.io/projected/04f8f9f1-158f-4462-8dd5-061f94507728-kube-api-access-54gkm\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.102347 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.102324 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f8f9f1-158f-4462-8dd5-061f94507728-openshift-service-ca\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.104196 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.104170 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-maas-api-tls\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.104332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.104310 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-metrics-certs\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.109219 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.109199 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54gkm\" (UniqueName: \"kubernetes.io/projected/04f8f9f1-158f-4462-8dd5-061f94507728-kube-api-access-54gkm\") pod \"maas-api-655f6fc5df-7znl5\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.257011 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.256940 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:39.537007 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.536977 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5"] Nov 01 16:48:39.538317 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:48:39.538287 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f8f9f1_158f_4462_8dd5_061f94507728.slice/crio-90b528812defce442f05669ef7fdd6acc0bb7ecff23666459a3ddbc6e59ad221 WatchSource:0}: Error finding container 90b528812defce442f05669ef7fdd6acc0bb7ecff23666459a3ddbc6e59ad221: Status 404 returned error can't find the container with id 90b528812defce442f05669ef7fdd6acc0bb7ecff23666459a3ddbc6e59ad221 Nov 01 16:48:39.539899 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.539874 4573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 16:48:39.946210 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:39.946176 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" event={"ID":"04f8f9f1-158f-4462-8dd5-061f94507728","Type":"ContainerStarted","Data":"90b528812defce442f05669ef7fdd6acc0bb7ecff23666459a3ddbc6e59ad221"} Nov 01 16:48:40.952170 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:40.952126 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" event={"ID":"04f8f9f1-158f-4462-8dd5-061f94507728","Type":"ContainerStarted","Data":"60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209"} Nov 01 16:48:40.952571 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:40.952205 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:40.968816 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:40.968744 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" podStartSLOduration=2.359438714 podStartE2EDuration="2.968730761s" totalImagesPullingTime="609.292047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:40.967628363 +0000 UTC m=+3658.670794298" watchObservedRunningTime="2026-11-01 16:48:40.968730761 +0000 UTC m=+3658.671896696" Nov 01 16:48:44.023728 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:44.023688 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2"] Nov 01 16:48:44.026495 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:44.026471 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-p8cm2"] Nov 01 16:48:44.456492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:44.456460 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21832286-b3c2-4059-b9ea-7cb04e87db22" path="/var/lib/kubelet/pods/21832286-b3c2-4059-b9ea-7cb04e87db22/volumes" Nov 01 16:48:45.826378 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:45.826342 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:46.274447 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:46.274414 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:46.373919 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:46.373876 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:46.963527 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:46.963502 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:47.207169 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.207139 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5"] Nov 01 16:48:47.207353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.207323 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" podUID="04f8f9f1-158f-4462-8dd5-061f94507728" containerName="maas-api" containerID="cri-o://60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209" gracePeriod=30 Nov 01 16:48:47.246437 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.246399 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7"] Nov 01 16:48:47.253295 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.253264 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.260611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.260585 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7"] Nov 01 16:48:47.383435 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.383402 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b6d1d8-5692-4468-b7f4-8d8a33247596-metrics-certs\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.383580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.383443 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b6d1d8-5692-4468-b7f4-8d8a33247596-openshift-service-ca\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.383719 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.383695 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxnbr\" (UniqueName: \"kubernetes.io/projected/b1b6d1d8-5692-4468-b7f4-8d8a33247596-kube-api-access-sxnbr\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.383831 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.383777 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b6d1d8-5692-4468-b7f4-8d8a33247596-maas-api-tls\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.477645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.477584 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:47.484853 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.484825 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxnbr\" (UniqueName: \"kubernetes.io/projected/b1b6d1d8-5692-4468-b7f4-8d8a33247596-kube-api-access-sxnbr\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.484991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.484869 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b6d1d8-5692-4468-b7f4-8d8a33247596-maas-api-tls\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.484991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.484902 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b6d1d8-5692-4468-b7f4-8d8a33247596-metrics-certs\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.484991 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.484915 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b6d1d8-5692-4468-b7f4-8d8a33247596-openshift-service-ca\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.485654 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.485624 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1b6d1d8-5692-4468-b7f4-8d8a33247596-openshift-service-ca\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.487353 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.487332 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b1b6d1d8-5692-4468-b7f4-8d8a33247596-maas-api-tls\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.487469 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.487368 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1b6d1d8-5692-4468-b7f4-8d8a33247596-metrics-certs\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.492818 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.492790 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxnbr\" (UniqueName: \"kubernetes.io/projected/b1b6d1d8-5692-4468-b7f4-8d8a33247596-kube-api-access-sxnbr\") pod \"maas-api-796985b8cc-r7lp7\" (UID: \"b1b6d1d8-5692-4468-b7f4-8d8a33247596\") " pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.585288 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.585255 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-maas-api-tls\") pod \"04f8f9f1-158f-4462-8dd5-061f94507728\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " Nov 01 16:48:47.585470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.585314 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f8f9f1-158f-4462-8dd5-061f94507728-kube-api-access-54gkm\" (UniqueName: \"kubernetes.io/projected/04f8f9f1-158f-4462-8dd5-061f94507728-kube-api-access-54gkm\") pod \"04f8f9f1-158f-4462-8dd5-061f94507728\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " Nov 01 16:48:47.585470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.585406 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f8f9f1-158f-4462-8dd5-061f94507728-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f8f9f1-158f-4462-8dd5-061f94507728-openshift-service-ca\") pod \"04f8f9f1-158f-4462-8dd5-061f94507728\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " Nov 01 16:48:47.585470 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.585426 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-metrics-certs\") pod \"04f8f9f1-158f-4462-8dd5-061f94507728\" (UID: \"04f8f9f1-158f-4462-8dd5-061f94507728\") " Nov 01 16:48:47.585809 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.585778 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f8f9f1-158f-4462-8dd5-061f94507728-openshift-service-ca" pod "04f8f9f1-158f-4462-8dd5-061f94507728" (UID: "04f8f9f1-158f-4462-8dd5-061f94507728"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:48:47.587422 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.587393 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-maas-api-tls" pod "04f8f9f1-158f-4462-8dd5-061f94507728" (UID: "04f8f9f1-158f-4462-8dd5-061f94507728"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:48:47.587524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.587435 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f8f9f1-158f-4462-8dd5-061f94507728-kube-api-access-54gkm" pod "04f8f9f1-158f-4462-8dd5-061f94507728" (UID: "04f8f9f1-158f-4462-8dd5-061f94507728"). InnerVolumeSpecName "kube-api-access-54gkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:48:47.587524 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.587444 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-metrics-certs" pod "04f8f9f1-158f-4462-8dd5-061f94507728" (UID: "04f8f9f1-158f-4462-8dd5-061f94507728"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 16:48:47.590643 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.590625 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:47.686852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.686815 4573 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04f8f9f1-158f-4462-8dd5-061f94507728-openshift-service-ca\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:47.686852 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.686855 4573 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-metrics-certs\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:47.687051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.686870 4573 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04f8f9f1-158f-4462-8dd5-061f94507728-maas-api-tls\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:47.687051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.686884 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54gkm\" (UniqueName: \"kubernetes.io/projected/04f8f9f1-158f-4462-8dd5-061f94507728-kube-api-access-54gkm\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:48:47.864817 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.864788 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7"] Nov 01 16:48:47.866594 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:48:47.866570 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1b6d1d8_5692_4468_b7f4_8d8a33247596.slice/crio-51c8dc651e2cfae0f6d2298f9ff2e8c84acf1d6ef9d7811828a9c9164524cb05 WatchSource:0}: Error finding container 51c8dc651e2cfae0f6d2298f9ff2e8c84acf1d6ef9d7811828a9c9164524cb05: Status 404 returned error can't find the container with id 51c8dc651e2cfae0f6d2298f9ff2e8c84acf1d6ef9d7811828a9c9164524cb05 Nov 01 16:48:47.992417 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.992383 4573 generic.go:356] "Generic (PLEG): container finished" podID="04f8f9f1-158f-4462-8dd5-061f94507728" containerID="60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209" exitCode=0 Nov 01 16:48:47.992888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.992429 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" Nov 01 16:48:47.992888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.992468 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" event={"ID":"04f8f9f1-158f-4462-8dd5-061f94507728","Type":"ContainerDied","Data":"60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209"} Nov 01 16:48:47.992888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.992526 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5" event={"ID":"04f8f9f1-158f-4462-8dd5-061f94507728","Type":"ContainerDied","Data":"90b528812defce442f05669ef7fdd6acc0bb7ecff23666459a3ddbc6e59ad221"} Nov 01 16:48:47.992888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.992549 4573 scope.go:122] "RemoveContainer" containerID="60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209" Nov 01 16:48:47.993851 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:47.993831 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" event={"ID":"b1b6d1d8-5692-4468-b7f4-8d8a33247596","Type":"ContainerStarted","Data":"51c8dc651e2cfae0f6d2298f9ff2e8c84acf1d6ef9d7811828a9c9164524cb05"} Nov 01 16:48:48.001608 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:48.001590 4573 scope.go:122] "RemoveContainer" containerID="60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209" Nov 01 16:48:48.001899 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:48:48.001882 4573 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209\": container with ID starting with 60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209 not found: ID does not exist" containerID="60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209" Nov 01 16:48:48.001968 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:48.001908 4573 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209"} err="failed to get container status \"60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209\": rpc error: code = NotFound desc = could not find container \"60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209\": container with ID starting with 60c0b861f96345d50c04baf45024fd8b5d910886087cece8eec158a63283f209 not found: ID does not exist" Nov 01 16:48:48.016193 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:48.016174 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5"] Nov 01 16:48:48.019829 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:48.019810 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-655f6fc5df-7znl5"] Nov 01 16:48:48.456301 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:48.456271 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f8f9f1-158f-4462-8dd5-061f94507728" path="/var/lib/kubelet/pods/04f8f9f1-158f-4462-8dd5-061f94507728/volumes" Nov 01 16:48:49.001646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:49.001608 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" event={"ID":"b1b6d1d8-5692-4468-b7f4-8d8a33247596","Type":"ContainerStarted","Data":"c038c8fc93110ed2f558c41fb6ab01a46598dabde1ea63fffc8f53c93a6718e7"} Nov 01 16:48:49.002105 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:49.001743 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:49.018633 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:49.018586 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" podStartSLOduration=1.608004662 podStartE2EDuration="2.018574325s" totalImagesPullingTime="410.569663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:49.017347931 +0000 UTC m=+3666.720513869" watchObservedRunningTime="2026-11-01 16:48:49.018574325 +0000 UTC m=+3666.721740261" Nov 01 16:48:49.093474 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:49.093437 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:49.192924 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:49.192889 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:50.261330 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.261293 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:50.295487 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.295456 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k"] Nov 01 16:48:50.295939 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.295925 4573 cpu_manager.go:414] "containerMap: removing container" podUID="04f8f9f1-158f-4462-8dd5-061f94507728" containerName="maas-api" Nov 01 16:48:50.296023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.295940 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f8f9f1-158f-4462-8dd5-061f94507728" containerName="maas-api" Nov 01 16:48:50.296023 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.296010 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f8f9f1-158f-4462-8dd5-061f94507728" containerName="maas-api" Nov 01 16:48:50.299438 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.299411 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:50.309340 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.309312 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f9b881a4\"" Nov 01 16:48:50.313602 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.313515 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k"] Nov 01 16:48:50.353716 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.353679 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:50.417762 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.417716 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k\" (UID: \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:50.417993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.417844 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72t5\" (UniqueName: \"kubernetes.io/projected/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-kube-api-access-w72t5\") pod \"payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k\" (UID: \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:50.452113 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.452077 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:50.518725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.518631 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k\" (UID: \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:50.518725 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.518704 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72t5\" (UniqueName: \"kubernetes.io/projected/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-kube-api-access-w72t5\") pod \"payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k\" (UID: \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:50.519361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.519335 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k\" (UID: \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:50.527090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.527067 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72t5\" (UniqueName: \"kubernetes.io/projected/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-kube-api-access-w72t5\") pod \"payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k\" (UID: \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:50.551803 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.551779 4573 scope.go:122] "RemoveContainer" containerID="afa65aeffde05c699142c3091408fa79b90978ba42890844e6ce76a2032b1bf9" Nov 01 16:48:50.626579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.626549 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:50.905017 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:50.904989 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k"] Nov 01 16:48:50.907092 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:48:50.907066 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c757ead_e87f_46e5_bbd8_05ba6f1efb27.slice/crio-ca76ce709c5e084d2634996238ea1e78fccab5b960fc7958dbed6af238e09ddc WatchSource:0}: Error finding container ca76ce709c5e084d2634996238ea1e78fccab5b960fc7958dbed6af238e09ddc: Status 404 returned error can't find the container with id ca76ce709c5e084d2634996238ea1e78fccab5b960fc7958dbed6af238e09ddc Nov 01 16:48:51.013534 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:51.013503 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" event={"ID":"3c757ead-e87f-46e5-bbd8-05ba6f1efb27","Type":"ContainerStarted","Data":"5b729387d02de2e8a85bc5d296b27b9235f315e2072ddefee5d5520f35dca902"} Nov 01 16:48:51.013689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:51.013562 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" event={"ID":"3c757ead-e87f-46e5-bbd8-05ba6f1efb27","Type":"ContainerStarted","Data":"ca76ce709c5e084d2634996238ea1e78fccab5b960fc7958dbed6af238e09ddc"} Nov 01 16:48:51.013689 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:51.013587 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:48:51.029364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:51.029322 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" podStartSLOduration=1.029308494 podStartE2EDuration="1.029308494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 16:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:48:51.028120778 +0000 UTC m=+3668.731286719" watchObservedRunningTime="2026-11-01 16:48:51.029308494 +0000 UTC m=+3668.732474430" Nov 01 16:48:51.102553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:51.102517 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:51.892516 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:51.892482 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:52.491564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:52.491523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:52.756956 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:52.756870 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:48:53.282739 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:53.282705 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:54.094426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:54.094394 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:54.364302 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:48:54.364268 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:48:54.825887 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:54.825805 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:55.015724 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:55.015694 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-796985b8cc-r7lp7" Nov 01 16:48:55.522471 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:55.522434 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:56.213768 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:56.213718 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:56.915144 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:56.915111 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:57.614606 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:57.614568 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:58.467558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:58.467529 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:48:59.212081 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:48:59.212051 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:00.033190 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:00.033151 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:00.775429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:00.775384 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:01.572087 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:01.572054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:02.022040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:02.022013 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:49:02.187175 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:02.187140 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:02.873318 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:02.873278 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:03.504446 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:03.504407 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:04.231095 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:04.231054 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:04.908580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:04.908541 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:05.707478 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:05.707422 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:06.505323 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:06.505256 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:07.159856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:07.159813 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:07.908630 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:07.908593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:08.597104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:08.597074 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:09.283420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:09.283383 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:09.889795 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:09.889734 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:10.028040 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:10.028000 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k"] Nov 01 16:49:10.075531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:10.075496 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:10.176055 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:10.175979 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:10.471488 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:10.471413 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:10.603407 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:10.603371 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:10.687558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:10.687515 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:11.875245 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:11.875212 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:49:15.030002 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.029949 4573 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" podUID="3c757ead-e87f-46e5-bbd8-05ba6f1efb27" containerName="payload-pre-processing" containerID="cri-o://5b729387d02de2e8a85bc5d296b27b9235f315e2072ddefee5d5520f35dca902" gracePeriod=25 Nov 01 16:49:15.145523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.145494 4573 generic.go:356] "Generic (PLEG): container finished" podID="3c757ead-e87f-46e5-bbd8-05ba6f1efb27" containerID="5b729387d02de2e8a85bc5d296b27b9235f315e2072ddefee5d5520f35dca902" exitCode=0 Nov 01 16:49:15.145651 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.145570 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" event={"ID":"3c757ead-e87f-46e5-bbd8-05ba6f1efb27","Type":"ContainerDied","Data":"5b729387d02de2e8a85bc5d296b27b9235f315e2072ddefee5d5520f35dca902"} Nov 01 16:49:15.277050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.277028 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:49:15.361678 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.361653 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-kube-api-access-w72t5\" (UniqueName: \"kubernetes.io/projected/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-kube-api-access-w72t5\") pod \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\" (UID: \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\") " Nov 01 16:49:15.361854 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.361706 4573 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-plugins-config-volume\") pod \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\" (UID: \"3c757ead-e87f-46e5-bbd8-05ba6f1efb27\") " Nov 01 16:49:15.362091 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.362067 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-plugins-config-volume" pod "3c757ead-e87f-46e5-bbd8-05ba6f1efb27" (UID: "3c757ead-e87f-46e5-bbd8-05ba6f1efb27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 16:49:15.363734 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.363710 4573 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-kube-api-access-w72t5" pod "3c757ead-e87f-46e5-bbd8-05ba6f1efb27" (UID: "3c757ead-e87f-46e5-bbd8-05ba6f1efb27"). InnerVolumeSpecName "kube-api-access-w72t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 16:49:15.462522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.462483 4573 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w72t5\" (UniqueName: \"kubernetes.io/projected/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-kube-api-access-w72t5\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:15.462522 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:15.462517 4573 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c757ead-e87f-46e5-bbd8-05ba6f1efb27-plugins-config-volume\") on node \"ip-10-0-141-145.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 16:49:16.151737 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:16.151712 4573 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" Nov 01 16:49:16.152164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:16.151772 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k" event={"ID":"3c757ead-e87f-46e5-bbd8-05ba6f1efb27","Type":"ContainerDied","Data":"ca76ce709c5e084d2634996238ea1e78fccab5b960fc7958dbed6af238e09ddc"} Nov 01 16:49:16.152164 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:16.151829 4573 scope.go:122] "RemoveContainer" containerID="5b729387d02de2e8a85bc5d296b27b9235f315e2072ddefee5d5520f35dca902" Nov 01 16:49:16.176365 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:16.176323 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:16.185251 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:16.185225 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k"] Nov 01 16:49:16.201848 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:16.201826 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f9b881a4-6f54c8697f-54m2k"] Nov 01 16:49:16.457140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:16.457064 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c757ead-e87f-46e5-bbd8-05ba6f1efb27" path="/var/lib/kubelet/pods/3c757ead-e87f-46e5-bbd8-05ba6f1efb27/volumes" Nov 01 16:49:17.391544 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:17.391473 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:17.811999 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:17.811922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:17.899584 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:17.899531 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:21.133575 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:21.133536 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:21.228632 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:21.228596 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:22.254554 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:22.254510 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:22.352302 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:22.352260 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:22.453492 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:22.453455 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:23.115130 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:23.115098 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:23.769432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:23.769403 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:24.185874 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:24.185841 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:49:24.363885 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:49:24.363844 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:49:24.421891 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:24.421860 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:25.104099 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:25.104066 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:25.716953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:25.716923 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:26.428741 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:26.428709 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:27.114556 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:27.114520 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:27.905473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:27.905436 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:28.610074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:28.610022 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:29.405425 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:29.405364 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:30.107096 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:30.107046 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:30.852034 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:30.852002 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:31.547410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:31.547376 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:32.301482 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:32.301452 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:32.881647 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:32.881613 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:33.243973 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:33.243891 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:33.493873 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:33.493836 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:34.099159 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:34.099124 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:34.785923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:34.785892 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:35.403356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:35.403319 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:36.076646 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:36.076568 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:36.316402 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:36.316369 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:37.048308 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:37.048277 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:37.831938 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:37.831904 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:38.506583 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:38.506549 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:39.201473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:39.201429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:39.847772 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:39.847719 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:39.946464 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:39.946430 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:40.613272 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:40.613224 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:41.188552 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:41.188519 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:41.786430 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:41.786399 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:42.382850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:42.382810 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:42.988367 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:42.988330 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:43.587525 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:43.587491 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:44.189107 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:44.189077 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:44.783859 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:44.783825 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:45.384413 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:45.384379 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:45.982050 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:45.982013 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:46.784878 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:46.784847 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:47.405667 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:47.405633 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:48.064203 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:48.064168 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:48.678553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:48.678523 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:49.280409 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:49.280374 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:49.878111 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:49.878076 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:50.558027 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:50.557993 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:50.578983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:50.578959 4573 scope.go:122] "RemoveContainer" containerID="012aef113fb884055f1f2ebe35b1b999030a200fa43d195effbab9f48c870a6c" Nov 01 16:49:51.093348 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:51.093313 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:51.692021 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:51.691967 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:52.305074 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:52.305002 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:52.888776 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:52.888721 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:53.561717 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:53.561679 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:54.099953 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:54.099920 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:54.364829 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:49:54.364790 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:49:54.691410 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:54.691301 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:55.288149 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:55.288116 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:55.892058 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:55.892007 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:56.609414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:56.609355 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:57.184521 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:57.184486 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:57.879284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:57.879251 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:58.487355 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:58.487328 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:59.086566 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:59.086533 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:49:59.700463 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:49:59.700430 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:00.291426 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:00.291391 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:01.089721 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:01.089683 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:01.761477 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:01.761443 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:02.599573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:02.599538 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:03.186576 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:03.186521 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:03.796199 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:03.796170 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:04.385249 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:04.385206 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:04.988039 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:04.988004 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:05.594685 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:05.594655 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:06.182187 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:06.182155 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:06.778987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:06.778956 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:07.379966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:07.379931 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:07.991806 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:07.991729 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:08.603216 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:08.603182 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:09.308198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:09.308160 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:09.891434 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:09.891402 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:10.591156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:10.591121 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:11.184618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:11.184587 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:11.800947 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:11.800916 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:12.486429 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:12.486396 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:13.088129 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:13.088095 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:13.690811 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:13.690776 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:14.391420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:14.391390 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:14.990553 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:14.990518 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:15.589944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:15.589907 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:16.191652 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:16.191622 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:16.794156 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:16.794122 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:17.601967 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:17.601934 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:18.236562 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:18.236526 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:18.876053 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:18.876021 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:19.492432 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:19.492401 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:20.092592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:20.092552 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:20.771779 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:20.771735 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:21.372392 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:21.372359 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:21.986178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:21.986139 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:22.604029 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:22.603994 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:23.190448 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:23.190414 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:23.787871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:23.787841 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:24.363787 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:50:24.363738 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:50:24.394238 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:24.394211 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:24.980498 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:24.980467 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:25.603450 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:25.603418 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:26.194242 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:26.194202 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:26.783051 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:26.783018 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:27.392177 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:27.392145 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:28.083560 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:28.083522 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:28.689067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:28.689033 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:29.029124 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:29.029039 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s"] Nov 01 16:50:29.032439 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:29.032413 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-wpw6s"] Nov 01 16:50:29.384201 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:29.384171 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:29.899483 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:29.899446 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:30.456322 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:30.456291 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29ea145-1600-4f8c-b46e-a9f298cd1659" path="/var/lib/kubelet/pods/a29ea145-1600-4f8c-b46e-a9f298cd1659/volumes" Nov 01 16:50:30.498543 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:30.498513 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:31.094341 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:31.094311 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:31.699024 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:31.698991 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:32.368778 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:32.368730 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:32.992371 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:32.992338 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:33.287981 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:33.287906 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:33.599109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:33.599029 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:34.201313 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:34.201280 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:34.796948 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:34.796915 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:35.414128 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:35.414096 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:36.081621 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:36.081593 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:36.774145 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:36.774113 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:37.390580 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:37.390548 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:38.083743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:38.083711 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:38.762610 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:38.762574 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:39.398545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:39.398510 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:40.097658 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:40.097620 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:40.809104 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:40.809073 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:41.446767 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:41.446730 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:42.081992 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:42.081955 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:42.691178 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:42.691141 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:43.284622 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:43.284583 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:43.894944 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:43.894912 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:44.492907 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:44.492872 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:44.657902 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:44.657869 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:45.090612 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:45.090580 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:45.788510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:45.788474 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:46.460394 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:46.460357 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:47.121523 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:47.121485 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:47.859972 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:47.859935 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:48.028797 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:48.028762 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt"] Nov 01 16:50:48.032212 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:48.032183 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-hr7jt"] Nov 01 16:50:48.456414 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:48.456384 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886b1450-86f7-4cdf-b704-c083f03bd8ff" path="/var/lib/kubelet/pods/886b1450-86f7-4cdf-b704-c083f03bd8ff/volumes" Nov 01 16:50:48.511408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:48.511374 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:49.217476 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:49.217439 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:49.998015 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:49.997975 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:50.798545 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:50.798514 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:51.464765 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:51.464719 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:52.290634 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:52.290600 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:53.057369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:53.057341 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:53.817133 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:53.817091 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:54.363770 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:50:54.363711 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:50:54.652019 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:54.651922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:55.509579 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:55.509539 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:56.446356 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:56.446321 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:57.395258 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:57.395216 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:58.248010 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:58.247978 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:50:59.130871 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:50:59.130836 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:00.227538 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:00.227498 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:01.071687 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:01.071655 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:01.939510 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:01.939474 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:02.725573 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:02.725535 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:03.563979 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:03.563950 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:04.318955 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:04.318922 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:05.125398 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:05.125368 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:05.830379 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:05.830346 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:06.016980 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:06.016952 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:06.168800 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:06.168762 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:11.499768 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:11.499721 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:11.743364 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:11.743333 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:11.887984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:11.887949 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:12.335506 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:12.335429 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:13.188788 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:13.188744 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:51:18.267850 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:18.267814 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:18.376731 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:18.376695 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:18.643923 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:18.643887 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:20.795198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:20.795165 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:51:22.955068 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:22.955021 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:24.363032 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:51:24.362950 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:51:25.797629 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:25.797597 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:51:26.293900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:26.293846 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:28.471261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:28.470857 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-v67pm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:51:28.471261 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:28.470945 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-v67pm" podUID="73cbf3fc-1566-4516-9a74-c7684e42fe01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:51:33.340388 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:33.340349 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:51:41.198257 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:41.198221 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:51:45.398775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:45.397857 4573 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-r9wbx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 16:51:45.398775 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:45.397941 4573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-r9wbx" podUID="fc68abf9-50c7-4707-bd62-64fe92c4d9f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 16:51:50.613969 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:50.613938 4573 scope.go:122] "RemoveContainer" containerID="6179743cb95528662aade88cc0054936fdce6a9f3bd3b36bccb206037715ee51" Nov 01 16:51:50.622846 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:51:50.622827 4573 scope.go:122] "RemoveContainer" containerID="e189c0c6e695719ea668689d24ebbf4c5a13c14f7c20eb92ba33f94a97db6019" Nov 01 16:51:54.363771 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:51:54.363721 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:52:08.701715 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:08.701678 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-gxdck"] Nov 01 16:52:09.546531 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:09.546494 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:52:09.638613 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:09.638575 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:52:19.574467 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:19.574437 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7847bd554f-lqsgd_31af0e80-54db-44bf-8849-289f596b400e/manager/0.log" Nov 01 16:52:20.828631 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:20.828600 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wh7xc_89878e6b-ba80-4311-a306-ba78b713d080/manager/0.log" Nov 01 16:52:21.958090 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:21.958058 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796985b8cc-r7lp7_b1b6d1d8-5692-4468-b7f4-8d8a33247596/maas-api/0.log" Nov 01 16:52:22.333044 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:22.332966 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-lnpbh_28b84bee-caa8-40d6-aaa3-ce08bd6798c5/revoke-keys/0.log" Nov 01 16:52:22.510810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:22.510781 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-dhl7r_5a0bc57b-b9f4-4877-b458-a1cd20830f23/revoke-keys/0.log" Nov 01 16:52:22.876386 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:22.876358 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-be80fab5-ncnd7_84a0dc41-c45a-4e39-a7ac-53eb0186e00d/revoke-keys/0.log" Nov 01 16:52:23.231453 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:23.231381 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-9t8zq_c9417c68-46f1-441f-a77b-c7943426bbd7/postgres/0.log" Nov 01 16:52:24.362979 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:52:24.362936 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:52:25.303578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:25.303548 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4j9f6_c0e421c1-a5b1-417a-a2a2-153403e5af56/kuadrant-console-plugin/0.log" Nov 01 16:52:25.736307 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:25.736283 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gxdck_18b562d7-dad3-49f7-aaf0-49c9a465f7f9/limitador/0.log" Nov 01 16:52:27.197127 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:27.197097 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6759f4f87c-t6l44_ce438594-887f-464e-b710-3502d00c4ead/kube-auth-proxy/0.log" Nov 01 16:52:27.948115 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:27.948079 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-gnmrk_c1e3f0a9-1576-4410-b17e-be1afb4b73a8/istio-proxy/0.log" Nov 01 16:52:28.535644 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:28.535592 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-r9wbx_fc68abf9-50c7-4707-bd62-64fe92c4d9f9/istio-proxy/0.log" Nov 01 16:52:28.870279 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:28.870247 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-v67pm_73cbf3fc-1566-4516-9a74-c7684e42fe01/istio-proxy/0.log" Nov 01 16:52:29.426640 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:29.426610 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-44k4f_e5127857-fa25-4c87-8958-69cfb960d73f/payload-pre-processing/0.log" Nov 01 16:52:30.939358 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:30.939325 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-jnxjv_2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c/main/0.log" Nov 01 16:52:31.097243 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:31.097209 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-mrk7c_eedaa33d-b796-42e7-8a32-403dc27c0221/main/0.log" Nov 01 16:52:32.001109 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:32.001074 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt_a0218c49-8d6d-4bb3-8cf7-10634e6c7a44/main/0.log" Nov 01 16:52:33.398443 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:33.398399 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:52:41.681618 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:41.681588 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7847bd554f-lqsgd_31af0e80-54db-44bf-8849-289f596b400e/manager/0.log" Nov 01 16:52:42.861270 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:42.861243 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-wh7xc_89878e6b-ba80-4311-a306-ba78b713d080/manager/0.log" Nov 01 16:52:43.515672 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:43.515646 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:52:43.528665 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:43.528634 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:52:43.528894 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:43.528876 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:52:43.540161 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:43.540137 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x4htf_4a228ff8-0200-4b6f-8948-9e351926a3b7/ovn-acl-logging/0.log" Nov 01 16:52:43.781408 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:43.781326 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-796985b8cc-r7lp7_b1b6d1d8-5692-4468-b7f4-8d8a33247596/maas-api/0.log" Nov 01 16:52:44.127563 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:44.127537 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-lnpbh_28b84bee-caa8-40d6-aaa3-ce08bd6798c5/revoke-keys/0.log" Nov 01 16:52:44.304743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:44.304716 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-dhl7r_5a0bc57b-b9f4-4877-b458-a1cd20830f23/revoke-keys/0.log" Nov 01 16:52:44.646985 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:44.646961 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-be80fab5-ncnd7_84a0dc41-c45a-4e39-a7ac-53eb0186e00d/revoke-keys/0.log" Nov 01 16:52:44.988670 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:44.988581 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-9t8zq_c9417c68-46f1-441f-a77b-c7943426bbd7/postgres/0.log" Nov 01 16:52:46.606983 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:46.606955 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4j9f6_c0e421c1-a5b1-417a-a2a2-153403e5af56/kuadrant-console-plugin/0.log" Nov 01 16:52:47.019183 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:47.019108 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gxdck_18b562d7-dad3-49f7-aaf0-49c9a465f7f9/limitador/0.log" Nov 01 16:52:48.290642 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:48.290612 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6759f4f87c-t6l44_ce438594-887f-464e-b710-3502d00c4ead/kube-auth-proxy/0.log" Nov 01 16:52:49.005220 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:49.005187 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-gnmrk_c1e3f0a9-1576-4410-b17e-be1afb4b73a8/istio-proxy/0.log" Nov 01 16:52:49.542416 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:49.542389 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-r9wbx_fc68abf9-50c7-4707-bd62-64fe92c4d9f9/istio-proxy/0.log" Nov 01 16:52:49.898861 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:49.898835 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-v67pm_73cbf3fc-1566-4516-9a74-c7684e42fe01/istio-proxy/0.log" Nov 01 16:52:50.454886 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:50.454856 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-44k4f_e5127857-fa25-4c87-8958-69cfb960d73f/payload-pre-processing/0.log" Nov 01 16:52:51.484966 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:51.484937 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-jnxjv_2f5d2cae-2c42-4c6f-9cf4-801fc3a67f4c/main/0.log" Nov 01 16:52:51.663906 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:51.663880 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-mrk7c_eedaa33d-b796-42e7-8a32-403dc27c0221/main/0.log" Nov 01 16:52:52.533836 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:52.533806 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-plxrt_a0218c49-8d6d-4bb3-8cf7-10634e6c7a44/main/0.log" Nov 01 16:52:54.363530 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:52:54.363448 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:52:57.021558 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:57.021517 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r"] Nov 01 16:52:57.024709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:57.024683 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-dhl7r"] Nov 01 16:52:58.456309 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:52:58.456278 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0bc57b-b9f4-4877-b458-a1cd20830f23" path="/var/lib/kubelet/pods/5a0bc57b-b9f4-4877-b458-a1cd20830f23/volumes" Nov 01 16:53:04.397485 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:53:04.397441 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:53:10.028611 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:53:10.028574 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh"] Nov 01 16:53:10.031805 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:53:10.031775 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-lnpbh"] Nov 01 16:53:10.456329 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:53:10.456296 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28b84bee-caa8-40d6-aaa3-ce08bd6798c5" path="/var/lib/kubelet/pods/28b84bee-caa8-40d6-aaa3-ce08bd6798c5/volumes" Nov 01 16:53:24.362237 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:53:24.362202 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:53:30.022628 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:53:30.022591 4573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7"] Nov 01 16:53:30.025236 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:53:30.025210 4573 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be80fab5-ncnd7"] Nov 01 16:53:30.457076 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:53:30.457040 4573 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a0dc41-c45a-4e39-a7ac-53eb0186e00d" path="/var/lib/kubelet/pods/84a0dc41-c45a-4e39-a7ac-53eb0186e00d/volumes" Nov 01 16:53:33.445005 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:53:33.444969 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:53:54.363733 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:53:54.363697 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:54:18.583185 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:18.583146 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:54:18.678335 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:18.678296 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:54:22.134140 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:22.134111 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2g5v9_81dc536f-2ef1-43d3-aa56-ccbe4ecf66b1/global-pull-secret-syncer/0.log" Nov 01 16:54:22.398207 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:22.398093 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" Nov 01 16:54:22.581730 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:22.581696 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-145.us-east-2.compute.internal_6555262b15c85369c5c728ca31cfdef1/haproxy/0.log" Nov 01 16:54:24.363198 ip-10-0-141-145 kubenswrapper[4573]: E1101 16:54:24.363114 4573 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-lsszq" containerName="extractor" Nov 01 16:54:31.325743 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:31.325712 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-4j9f6_c0e421c1-a5b1-417a-a2a2-153403e5af56/kuadrant-console-plugin/0.log" Nov 01 16:54:31.430607 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:31.430584 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-gxdck_18b562d7-dad3-49f7-aaf0-49c9a465f7f9/limitador/0.log" Nov 01 16:54:32.528098 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.528065 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37931145-6e82-4af7-9dcb-32e8bb52146f/alertmanager/0.log" Nov 01 16:54:32.552732 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.552706 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37931145-6e82-4af7-9dcb-32e8bb52146f/config-reloader/0.log" Nov 01 16:54:32.578810 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.578736 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37931145-6e82-4af7-9dcb-32e8bb52146f/kube-rbac-proxy-web/0.log" Nov 01 16:54:32.604147 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.604126 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37931145-6e82-4af7-9dcb-32e8bb52146f/kube-rbac-proxy/0.log" Nov 01 16:54:32.633550 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.633528 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37931145-6e82-4af7-9dcb-32e8bb52146f/kube-rbac-proxy-metric/0.log" Nov 01 16:54:32.662473 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.662449 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37931145-6e82-4af7-9dcb-32e8bb52146f/prom-label-proxy/0.log" Nov 01 16:54:32.682137 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.682112 4573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq"] Nov 01 16:54:32.682537 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.682522 4573 cpu_manager.go:414] "containerMap: removing container" podUID="3c757ead-e87f-46e5-bbd8-05ba6f1efb27" containerName="payload-pre-processing" Nov 01 16:54:32.682582 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.682539 4573 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c757ead-e87f-46e5-bbd8-05ba6f1efb27" containerName="payload-pre-processing" Nov 01 16:54:32.682645 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.682635 4573 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c757ead-e87f-46e5-bbd8-05ba6f1efb27" containerName="payload-pre-processing" Nov 01 16:54:32.685994 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.685977 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.692278 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.692236 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c79r4\"/\"kube-root-ca.crt\"" Nov 01 16:54:32.693782 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.693762 4573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c79r4\"/\"default-dockercfg-fqmbm\"" Nov 01 16:54:32.697093 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.697067 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq"] Nov 01 16:54:32.699740 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.699725 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_37931145-6e82-4af7-9dcb-32e8bb52146f/init-config-reloader/0.log" Nov 01 16:54:32.769639 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.769613 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-proc\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.769827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.769657 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-sys\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.769827 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.769738 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w65gv\" (UniqueName: \"kubernetes.io/projected/5b684114-e15d-4702-85bb-480fa1d62359-kube-api-access-w65gv\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.769920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.769829 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-podres\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.769920 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.769859 4573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-lib-modules\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.790218 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.790160 4573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c79r4\"/\"openshift-service-ca.crt\"" Nov 01 16:54:32.870529 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870496 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-proc\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.870660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870548 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-sys\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.870660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870597 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w65gv\" (UniqueName: \"kubernetes.io/projected/5b684114-e15d-4702-85bb-480fa1d62359-kube-api-access-w65gv\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.870660 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870626 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-proc\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.870915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870671 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-podres\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.870915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870687 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-sys\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.870915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870704 4573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-lib-modules\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.870915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870794 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-podres\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.870915 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.870840 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b684114-e15d-4702-85bb-480fa1d62359-lib-modules\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.878557 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.878534 4573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w65gv\" (UniqueName: \"kubernetes.io/projected/5b684114-e15d-4702-85bb-480fa1d62359-kube-api-access-w65gv\") pod \"perf-node-gather-daemonset-229zq\" (UID: \"5b684114-e15d-4702-85bb-480fa1d62359\") " pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:32.887686 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.887669 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-855bfdd46d-trf7z_f008bfb6-889a-4049-bbb4-a0eb73f9cba0/metrics-server/0.log" Nov 01 16:54:32.915624 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.915604 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-zzpzm_565fe20a-c405-4502-8bf1-3bafaa7dc4c3/monitoring-plugin/0.log" Nov 01 16:54:32.949067 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.949046 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6dxbr_312b0901-ccbf-46ea-952e-2c36151d6122/node-exporter/0.log" Nov 01 16:54:32.974102 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:32.974086 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6dxbr_312b0901-ccbf-46ea-952e-2c36151d6122/kube-rbac-proxy/0.log" Nov 01 16:54:33.001198 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.001179 4573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:33.006374 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.006353 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6dxbr_312b0901-ccbf-46ea-952e-2c36151d6122/init-textfile/0.log" Nov 01 16:54:33.276077 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.275842 4573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq"] Nov 01 16:54:33.278245 ip-10-0-141-145 kubenswrapper[4573]: W1101 16:54:33.278211 4573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b684114_e15d_4702_85bb_480fa1d62359.slice/crio-a42314b3a4758ca8b6981c5539fdd38d4510b384d782587b8b8174978ff7a66f WatchSource:0}: Error finding container a42314b3a4758ca8b6981c5539fdd38d4510b384d782587b8b8174978ff7a66f: Status 404 returned error can't find the container with id a42314b3a4758ca8b6981c5539fdd38d4510b384d782587b8b8174978ff7a66f Nov 01 16:54:33.291984 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.291965 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/0.log" Nov 01 16:54:33.293351 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.293334 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/prometheus/1.log" Nov 01 16:54:33.313987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.313924 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/config-reloader/0.log" Nov 01 16:54:33.339332 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.339314 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/thanos-sidecar/0.log" Nov 01 16:54:33.366912 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.366874 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/kube-rbac-proxy-web/0.log" Nov 01 16:54:33.397454 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.397434 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/kube-rbac-proxy/0.log" Nov 01 16:54:33.426564 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.426541 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/kube-rbac-proxy-thanos/0.log" Nov 01 16:54:33.454369 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.454352 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d402a470-6adc-4cd1-b435-d2934102977d/init-config-reloader/0.log" Nov 01 16:54:33.511300 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.511275 4573 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cvct9_f7e5339a-b073-4184-9acc-5a701276ea1c/konnectivity-agent/0.log" limit=1024 Nov 01 16:54:33.579026 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.578969 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6755758fcb-2nfbk_8174b90f-0f97-4b31-9756-faa476cfcbb8/telemeter-client/0.log" Nov 01 16:54:33.615888 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.615869 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6755758fcb-2nfbk_8174b90f-0f97-4b31-9756-faa476cfcbb8/reload/0.log" Nov 01 16:54:33.640458 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.640439 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6755758fcb-2nfbk_8174b90f-0f97-4b31-9756-faa476cfcbb8/kube-rbac-proxy/0.log" Nov 01 16:54:33.776900 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.776863 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" event={"ID":"5b684114-e15d-4702-85bb-480fa1d62359","Type":"ContainerStarted","Data":"a77a991bb99784a66d5b51ebf40709673cec7cd485301aee128b0e0b2a32b521"} Nov 01 16:54:33.777118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.776932 4573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" event={"ID":"5b684114-e15d-4702-85bb-480fa1d62359","Type":"ContainerStarted","Data":"a42314b3a4758ca8b6981c5539fdd38d4510b384d782587b8b8174978ff7a66f"} Nov 01 16:54:33.777118 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.776961 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:33.794993 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:33.794945 4573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" podStartSLOduration=1.794923983 podStartE2EDuration="1.794923983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 16:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 16:54:33.792675921 +0000 UTC m=+4011.495841857" watchObservedRunningTime="2026-11-01 16:54:33.794923983 +0000 UTC m=+4011.498089927" Nov 01 16:54:38.109208 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:38.109179 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gx6wg_370dbf42-367c-48ac-abb8-86c01bfe2b54/dns/0.log" Nov 01 16:54:38.132592 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:38.132568 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gx6wg_370dbf42-367c-48ac-abb8-86c01bfe2b54/kube-rbac-proxy/0.log" Nov 01 16:54:38.213578 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:38.213555 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8fxdr_a22ea5ba-32e9-4552-802d-d11594d04f29/dns-node-resolver/0.log" Nov 01 16:54:38.979943 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:38.979912 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rsjf4_edc57054-ce41-4e4e-97f6-b127bf151d23/node-ca/0.log" Nov 01 16:54:39.791856 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:39.791826 4573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c79r4/perf-node-gather-daemonset-229zq" Nov 01 16:54:43.714964 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:43.714933 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6759f4f87c-t6l44_ce438594-887f-464e-b710-3502d00c4ead/kube-auth-proxy/0.log" Nov 01 16:54:43.855987 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:43.855955 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-gnmrk_c1e3f0a9-1576-4410-b17e-be1afb4b73a8/istio-proxy/0.log" Nov 01 16:54:43.971284 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:43.971209 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-r9wbx_fc68abf9-50c7-4707-bd62-64fe92c4d9f9/istio-proxy/0.log" Nov 01 16:54:44.044420 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:44.044390 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-v67pm_73cbf3fc-1566-4516-9a74-c7684e42fe01/istio-proxy/0.log" Nov 01 16:54:44.175361 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:44.175332 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-44k4f_e5127857-fa25-4c87-8958-69cfb960d73f/payload-pre-processing/0.log" Nov 01 16:54:46.357158 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:46.357128 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6759f4f87c-t6l44_ce438594-887f-464e-b710-3502d00c4ead/kube-auth-proxy/0.log" Nov 01 16:54:46.437072 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:46.437040 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-gnmrk_c1e3f0a9-1576-4410-b17e-be1afb4b73a8/istio-proxy/0.log" Nov 01 16:54:46.495709 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:46.495676 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-r9wbx_fc68abf9-50c7-4707-bd62-64fe92c4d9f9/istio-proxy/0.log" Nov 01 16:54:46.536971 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:46.536944 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-v67pm_73cbf3fc-1566-4516-9a74-c7684e42fe01/istio-proxy/0.log" Nov 01 16:54:46.569712 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:46.569686 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-b7q5z_8b9e4381-b8a2-4d0d-ade3-75df4d07810f/serve-healthcheck-canary/0.log" Nov 01 16:54:46.588843 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:46.588823 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-44k4f_e5127857-fa25-4c87-8958-69cfb960d73f/payload-pre-processing/0.log" Nov 01 16:54:47.312375 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:47.312333 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lsszq_da994e56-8f1d-4f00-893a-670984816adc/kube-rbac-proxy/0.log" Nov 01 16:54:47.336786 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:47.336738 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lsszq_da994e56-8f1d-4f00-893a-670984816adc/exporter/0.log" Nov 01 16:54:47.361405 ip-10-0-141-145 kubenswrapper[4573]: I1101 16:54:47.361378 4573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-lsszq_da994e56-8f1d-4f00-893a-670984816adc/extractor/0.log"